cemerick

Update 2011-11-08T01:41  We did it! Details and acknowledgements for all who donated now here!

About a year ago, I had the distinct privilege of having a front-row seat when the generosity of the Clojure community (and others!) helped Anthony Grimes (aka Raynes) attend the 2010 Clojure Conj.  Witnessing the success of that process made me certain that I would attempt to coordinate another “scholarship” fundraising drive for this year’s Clojure Conj.

Like Anthony last year, I wanted to help someone that:

1. had already contributed significantly to the Clojure community,
2. clearly has great potential, but
3. without some financial assistance, would not be able to attend the Conj and benefit from the meeting-of-the-minds that such conferences uniquely enable.

Over the summer, it became clear to me that Ambrose Bonnaire-Sergeant was that someone.  A computer science undergraduate at the University of Western Australia in Perth, Ambrose has distinguished himself in the Clojure community this year.

His introduction to many was PragPub magazine publishing his Growing a DSL with Clojure article in July.  In the article, Ambrose led us through the building of a simplified shell-scripting DSL in Clojure modeled on the open source stevedore library that he’s contributed to.  More recently, Ambrose has been busy hacking away at significant contributions to the core.logic (logic programming) and core.match (optimized pattern matching and predicate dispatch) libraries led by David Nolen, producing content like his Logic-Starter documentation project that provides a compelling logic programming primer using core.logic as the raw materials, and helping out with jark tooling.  Finally, his has been a consistently helpful voice on the main Clojure mailing list.

Now, getting Ambrose to the Conj is going to be somewhat more challenging than last year’s effort: he’s in Australia after all, which makes both the travel and lodging expenses quite steep.  To make matters even more interesting, flights from Perth to Raleigh/Durham seem to fluctuate in price significantly — from $2,500 to $3,500, depending on when and where I check them.

Thankfully, Ambrose did submit a talk proposal to the Conj — an Introduction to Logic Programming with Clojure — which was accepted!  This means that the Conj organizers will be picking up his lodging during the conference, as well as generously providing $1,000 to help defray Ambrose’s travel expenses.  Thus, we (only?) need to raise enough to cover the remainder of his flight, as well as two extra hotel nights (so he can recuperate from jet lag enough to deliver his talk without experiencing a sudden bout of narcolepsy!).

To be safe, I am therefore setting the fundraising goal to $3,000.  As long as the gods of airline pricing are not against us, this will safely cover the remainder of Ambrose’s expenses, allow him to attend the Conj, and allow you to enjoy his sure-to-be-enlightening introductory logic programming talk.  Even moreso than last year, it’s a sizable sum; but, again, I think this is a worthy cause, and I suspect the Clojure community will rise to meet the challenge.

(If any funds remain, they will be used to provide discounted admission to the Conj for other students.)

I’ve set up a PayPal donation button below, along with a progress bar.  Some FYIs:

• I’m just reusing Snowtide‘s PayPal account for this; so, you’ll be seeing its logo through the checkout process and “snowtide” on your credit card statement.
• I’ll be manually updating the progress bar, so don’t expect to see it go up immediately after donating!
• I’ll shut down the donation button once we hit the $3,000 target.
• I’ll add your name to this page as thanks, unless you notify me otherwise, either (preferably) in the comments associated with your PayPal transaction or by email/twitter/irc.

Finally, as an added incentive: at the end of the fundraiser, the three people with the largest donations will each receive a copy of my forthcoming Clojure book, Clojure Programming from O’Reilly, signed by me (and probably my co-conspirators as well, Christophe Grand and Brian Carper).

If you have any questions at all, leave a comment here, or don’t hesitate to contact me if it your question involves confidential details (payment issues, etc).

We did it!

Details and acknowledgements for all who donated now here!

A few weeks ago, I put together a flowchart to help choose the right type-definition form in Clojure in various circumstances.

Afterwards, OGINO Masanori contacted me, offering to produce a Japanese translation of the flowchart.  The result looks great…thank you, Masanori!  We both hope it is useful to all Japanese Clojure programmers:

A few weeks ago, I opened the 2011 State of Clojure survey.  As with last year’s survey, my aim was to take a snapshot of the Clojure community — our origins in aggregate, how and where we are using Clojure, and what needs to improve to help the community grow and prosper.

First, some facts about the survey and how it was conducted. It was available for approximately 5 days (Wednesday night through Monday afternoon), during which time results were not available.  The survey was announced primarily via Twitter and two messages to the main Clojure mailing list.  670 responses were received — which, for sake of comparison, is ~13% of the ~5,000 subscribers to that mailing list.  As I’ve said before, I’m not a statistician, nor am I a professional when it comes to surveying, polling, or data collection of this sort in general, but this would seem to be a very fair sampling of the Clojure community.

With that, I’m going to run through each question in the survey, and offer my own perspective on each question’s results.  In contrast to last year’s results post, I’ll skip the Clojure cheerleading: there were certainly oodles of glowing comments of how fantastic Clojure is, but I think it’s safe to take that as a baseline at this point, and talk more about how things can be improved. :-)

See the link at the end to the raw data to satisfy all your statistical urges.

(Note: Any question whose results add up to more than 100% allowed respondents to select more than one response.)

Q: How long have you been using Clojure?

I think it’s safe to say that the Clojure community is growing (certainly more subscribers to the ML, more respondents to this survey than last year’s, more companies hiring Clojure programmers, etc), but trying to more precise than that would be folly.

One thing to note is that, nearly half of all respondents have been using Clojure for less than a year.  Last year had similar relative numbers of newcomers, but we aren’t seeing a big bulge of people reporting that they’ve been using Clojure for a year here.  This tell me that we’re doing a good job of attracting people to try Clojure, but they’re dropping out after some period of experimentation.  Surely some of that is natural, but some of that attrition can likely be attributed to the state of the resources provided to help people get started, and then get productive.  More about that later.

Q: How would you characterize your use of Clojure today?

Compared to last year’s survey results, there’s been an 8-point bump in the proportion of respondents that use Clojure at work, and an 8-point bump in the proportion that use it in school.  Though tinkering and hobby projects aren’t to be dismissed, this rise in “serious” use of Clojure is a very good sign — more people are feeling more comfortable using it the most important areas of their life.

Q: What is the status of Clojure in your workplace?

As was the case last year, 50% of respondents don’t use Clojure at work, but the rest of the numbers are pretty compelling: 31% use it at work in an “endorsed” capacity (up from 23% last year), and 22% of that usage is in production settings.  Again, signs that Clojure is being relied upon more, and perhaps with relatively little pushback from management given the low 9% rate of under-the-radar Clojure usage.

Q: In which domain(s) are you using Clojure?

This chart is nearly identical to the results from last year.  Web development and math/data analysis remain ubiquitous; lots of people are contributing to open source Clojure projects, and nonrelational databases remain preferred by Clojure programmers over RDBMS.

I enjoyed reading through the “other” domains people specified, including:

• Unmanned Vehicle Control
• AI / Natural Language Processing
• Geoinformatics / GIS
• semantic web
• Rapid prototyping of ideas that are later ported to iOS. (Is this common?)
• music via Overtone

The first two domains means that the robot invasion is coming, and they’ll all be running Clojure.

Q: Which development environment(s) do you use to work with Clojure?

This is also nearly identical to the results from last year, with mostly minor moves.  Emacs continues its dominance, though Vim + vimclojure did tick up to 21% from 18% last year.  All the other environments retained the same levels of support, except for the Netbeans + Enclojure combination, which dropped from 13% last year to 5% this year.

I remain baffled by the significant usage of command-line REPLs.  Just not my cup of tea, I suppose.

Honorable mentions in the “other” column include Sublime Text 2, JEdit, and Bluefish.  Also:

MCLIDE…should be a model for everyone.

Q: Clojure is primarily a JVM-hosted language. Which other platform(s) would you be interested in using Clojure on, given a mature implementation?

First, a couple of quotes from the “other” column:

None, JVM is enough for me.

and:

None. You have better things to do.

Compared to last year’s survey, CLR/.NET and Cocoa dropped quite a bit in relative terms, probably due to my adding a separate LLVM option.

Honorable mentions include Parrot, Erlang/BEAM, and Guile.

Those that want Clojure for the CLR should go check out ClojureCLR, tirelessly maintained by David Miller.  I honestly don’t know what its status is, although I have heard that some people are using it in serious applications these days.  If it is indeed “mature”, I apologize for implying that it’s not by including it in the question.

I’m inclined to drop this question from future surveys.  It’s sort of like asking, “What color unicorn would you like to own?” The implementation of other backends for Clojure is an incredibly difficult undertaking, and will likely be done very quietly by individuals or very small teams aiming to solve persistent problems of their own, not in response to an anonymous show of hands.

Q: What language did you use just prior to adopting Clojure — or, if Clojure is not your primary language now, what is that primary language?

As in last year’s survey, Clojure programmers indicate that they “come from” Java, Ruby, and Python, in that order and in roughly the same distribution.  Everything else is background noise.

Q: If Clojure disappeared tomorrow, what language(s) might you use as a “replacement”?

A quote from someone’s “other” response:

There is no replacement.  Please, do not disappear.

The winners here remain constant from last year.  Clojure programmers really like functional programming, and enough of them like the JVM enough to jump to Scala or JRuby if necessary, even though that’s not where they came from.

Honorable mentions include Go, Factor, F#, and Racket.

Q: What have been the biggest wins for you in using Clojure?

Various miscellaneous thoughts:

• As much as some people gripe about the JVM, a far larger number are benefiting from it; ~60% noting it as a “big win” is a helluva endorsement of the choice of the JVM as a host platform.
• People love their functional programming.  In fact, if you look at where Clojure programmers came from, and where they would go to if Clojure were to disappear (see the prior two questions), it seems clear that Clojure is a gateway drug for FP.
• It seems that relatively few people are taking advantage of some of Clojure’s most sophisticated and unique features: metadata; protocols, records, and types; and multimethods.  These facilities are absolutely game-changers, at least IMO.  Either most domains have no use for them (I can’t believe that), or most people don’t know how to use them effectively, thus they are left unused.  Those of us that write about and teach Clojure, take note.

Q: What has been most frustrating for you in your use of Clojure; or, what has kept you from using Clojure more than you do now?

A lot of the most common and most frustrating problems were expanded upon in the free-form responses, so I talk about them a bit there (see below). Otherwise, just a couple thoughts:

• Very few people have found the Clojure community to be a problem — just 2% report having unpleasant community interactions.  That’s both unsurprising and huge.  However, rather than simply bask in the awesomeness of the Clojure community, let’s ask: is it possible to make that number be 0% next year?  What can we do to minimize those negative interactions?  Are they caused by random misunderstandings, or are there just a few bad apples?
• While there’s been a lot of third-party projects aiming to improve the Clojure documentation landscape (both ClojureDocs and Clojure Atlas received positive mentions), people remain unhappy with the state of Clojure’s documentation, tutorials, and other resources.  Last year’s survey didn’t ask this question, so we have no point of comparison, but my impression is that the number of complaints in this department has stayed steady.

Q: What do you think is Clojure’s most glaring weakness / blind spot / problem?

This is where most of the fun is: a free-form text entry field that respondents could use to vent their frustrations — and, of course, they did!  The topics named in the responses to this question ranged all over the place: from irritations about particular libraries to issues with outdated blog-documentation, people didn’t hesitate to express themselves.  If you’d like, go read what people had to say directly (see the raw data link below).  Here, I’ll run through the most common complaints I saw.

(FYI, the amount of attention I give to each issue is not proportional to the number of people that raised it; I just happen to find some issues more interesting to write about than others.)

The JVM

Some people love that Clojure uses the JVM; others, though, quite simply despise it.  Specific complaints range from its typical startup time (particularly painful for those wanting to use Clojure on the command line) to “leakage” of Java build and dependency management processes (i.e. all Clojure build tools reuse the Maven dependency model and many build on top of Maven itself) to difficulty using native libraries.  A vocal minority of Clojure programmers (at least within this survey) wish dearly to be able to use Clojure without the “baggage” of the JVM.

Personally, I am quite at home with the JVM, so this doesn’t resonate with me, but I can empathize.  The choice of the JVM was a practical one (discussed in the Languages and Platforms section here).  With the lone exception of the CLR/.NET, there’s really no other environment out there that you could reasonably build a language on top of and get a comparable mix of mature VM implementations, existing libraries, community and tool support, and so on…but, if you don’t like the VM options, the existing libraries, and have little use for such tool support, then the JVM is likely to feel like a 400-lb. high-maintenance gorilla you need to carry with you whereever you go.

That said, without its JVM underpinnings, I really doubt that we would see Clojure being accepted into commercial environments as we have been (note the figures above of people using Clojure at work), or supported by platforms like Heroku, Amazon Web Services, and so on.  The size of the sandbox you play in does matter — otherwise, we’d have seen broad third-party tool and platform support for Common Lisp and various Scheme implementations years ago.

No “Clojure Starter Kit”

People continue to ache for an easy way to get started with Clojure, a sane development environment, and the libraries they need.  The issues involved here are legion, and they’re all hairy: platform-native launchers, the pains of emacs/SLIME and/or the various IDEs, integrated documentation (with cross-referenced usage examples), tracking of library status (github forks + the deployment of personal artifacts to Clojars makes for much confusion), modern debugger support, and so on.

Some of these things have gotten better over the last year, some haven’t.  A couple of highlighted issues that are current pain points:

The default REPL sucks

I’ll agree with this one wholeheartedly.  Clojure’s default REPL, run via a plain-vanilla command line, is just really unpleasant.  It should ship with readline/jline out of the box, and some suggested (reasonably, IMO) that some form of tab completion would be most welcome.  This is something that other languages (JRuby and Groovy come to mind) have taken care of very nicely.

IDE desperation

As much as most people love their emacs, there is a significant number that, even if they use emacs currently, wish dearly for a “regular” IDE for their Clojure programming.  Eclipse is mentioned most often, followed by IntelliJ; both have plugins — Counterclockwise and La Clojure, respectively — but they are apparently not up to snuff for many respondents.

As a contributor to Counterclockwise, I can see how that could be the case.  Many enhancements to it have been made in the past year, to the point where I switched to it full-time sometime late last summer, but even my personal wishlist there is long.

In any case, it’s clear that more needs to be done in this area so that high-quality Clojure tooling is easily accessible to all.

Error messages

This is also a reprise from last year.  The Clojure stack is the JVM stack (a very good thing if you care about performance at all!), but that means that Clojure stack traces are fundamentally JVM stack traces.  This means that you’ll likely see a lot of intervening stack frames from Java code when you print a Clojure stack trace.  Many people find this confusing, especially beginners.

There are a number of libraries for helping with the readability of stack traces, including clj-stacktrace and stacktrace utilities included with Clojure in the clojure.stacktrace and clojure.repl namespaces. There is ongoing debate about how best to skin stacktraces to maximize their usefulness.

Separate from stack traces are the issue of compiler error messages, which was a big pain point last year.  I don’t see that same magnitude of complaints in this area; if I remember correctly, some effort was made to improve error messages emitted by Clojure’s compiler over the past year.  Perhaps those improvements were sufficient to alleviate the prior pains.

Debuggers

No matter what tooling you use with Clojure, there is no doubt that current debuggers need further improvement to meet peoples’ expectations.  While the excellent CDT is out there, there seems to be a fair bit of configuration pain associated with it.  And, I can’t be certain, but I suspect that requests for a capable debugger made via the survey may correllate strongly with responses wishing for better non-emacs development environments — making me think that “we need a better debugger” should really be taken to mean “we need a better point-and-click debugger”, with the sort of visually interactive data structure inspectors that Eclipse and Visual Studio have made de rigueur over the years.

“Parens still scare the crap out of people”

A number of respondents indicated that Lisp, as a basic concept (and in its typical parenthetical manifestation, i.e. “oatmeal with fingernail clippings mixed in”) remains an obstacle to Clojure’s continued adoption.

I don’t actually believe this.  Yes, I’m sure there’s plenty of people who react poorly to seeing Clojure (or any Lisp) at first, whether due to simple aversion to the unfamiliar, cargo-cultism, or lingering memories of pain associated with “old world” Lisps.  However, the first will be convinced by education, facts, and results (currently being delivered, based on the workplace usage stats above and recent Clojure-related business news), and the second will be convinced by the first.  I’m guessing the third group is small enough to not be a long-term problem.

Fundamentally, Clojure either is or is not a effective option for building compelling software.  If it is, then every successful, ideally, astonishing Clojure app that gets out in the world will convince some number that Clojure is something to consider for their efforts.  Everything else that matters flows from this.

Public relations

Where the prior topic might be considered the “public relations of the language”, this one is related to public relations for Clojure, the project.  This is an area that is ripe for bikeshedding, so it can be tough to separate the wheat from the chaff when it comes to suggestions and complaints, but I’ll take a stab at it (and do some bikeshedding of my own, I suppose!).

(I can hear the curmudgeon technologists’ eyes rolling already, but this stuff is actually important if you want Clojure to succeed broadly.  If you’d prefer Clojure to be a protected sanctuary for the 1337, by all means, let us know by arguing against efforts to promote Clojure.  Cheers.)

Project status and activity

I’ll quote one comment in its entirety:

Clojure is awesome.  So is the community.  The website should reflect a burgeoning vibrant community and today, it just doesn’t do that.  No news, no changes or updates, the copyright is from 2010.  The one job on the jobs page has been there for a while.  According to the Wayback Time Machine, the main page hasn’t been updated this year, and only twice (aug and dec) last year. That’s kinda depressing. Compare this with scala-lang.org, which is full of activity – case studies, news feeds and flare.  The Clojure.org website could have twitter feeds, google groups digests, a road map, some case studies… Even if the news was “Hey, we’re still here, super busy on 1.3…” it would be better than no updates.  Clojure.com has a lot going for it, perhaps Clojure.org should be a community link…

Showcasing some marquee projects, such as Cascalog and Incanter, interviews posted on the site, guest bloggers, tutorials, a forum would be great.

The scala-lang.org comparison is appropriate, but hardly unique; comparing clojure.org to any of erlang.org, jruby.org, or jython.org gives a similar impression to the passer-by of a site (and therefore, project?) that is less active.

Of course, this is all about perception, and instant, blink-of-an-eye perception at at that.  There’s tons of stuff going on that impacts the language itself, as one can see by checking out the development wiki’s dashboard stream, or the project’s JIRA activity stream, for example.  There’s no disputing the actual state of affairs with regard to activity.  Rather, the question is: why not surface those flashes of activity (perhaps along with e.g. feeds from Planet Clojure and #clojure tweets) so that people who wander into clojure.org can immediately have a positive impression, rather than requiring of them an analytical comprehension of Clojure’s minutiae?

The jobs page smarts in particular, especially since there are lots of Clojure jobs available. The jobs page notes that it is “provided as a courtesy to those companies who have supported Clojure“, but I wouldn’t bet on the average visitor taking note of that distinction before mashing the back button.  Indicators like this can hurt a new(ish) language looking to maximize its reach, doubly so when the indicator is false.

Insofar as Clojure’s web presence is supported entirely by disparate (hard-working, invaluable!) volunteers as they are willing and able, these sorts of issues are likely to persist.  My instinct is that they could be fixed pretty easily, but it wants for a single person with sufficient integrated technical, marketing, and design sense and the authority to drive the Clojure web presence effectively for all likely classes of visitors.

“What has Rich Hickey done for me today?”

That’s not a real quote, but my semi-exaggerated tongue-in-cheek summation of various comments: not just about Rich, but about all manner of Clojure specialists (I think one comment said “Clojure old-timers”!) who stood a post at the mailing list and in #clojure irc, seemingly constantly for a number of years doing their best to welcome newcomers, explain the smallest of details, defuse flamewars, and generally set the tone of the community.

The thing about those sorts of activities is that, as long as someone is participating in conversation, they’re not getting things done.  Over the past year, we’ve definitely seen some of the most senior Clojure programmers in the community “pull back” to some extent as they go about their business, putting Clojure to work building apps, building companies (and writing books!), and, yes, feverishly working on Clojure tools, libraries, and Clojure itself.

Aside from the fact that you can no longer pepper Rich with questions in irc whenever you feel like it (a facility that I’m sure I overused all on my own back in the day!), one concrete consequence of this is that the tenor of the community must maintain and evolve on its own to a larger extent than it did in years one, two, and three (it had to happen sometime!).  I think #clojure irc has done that, with a number of consistently friendly, helpful, knowledgeable people stepping up to help.  I personally think the mailing list hasn’t fared as well — it’s grown much, much larger than it was in the early days, and without a universally-accepted authoritative voice, it’s far easier for threads to go into the weeds than irc conversations.  Members of Clojure/core have been involved here to some extent (Aaron, Alan, and the Stuarts come to mind), but I think the medium is largely against them.

Another facet of this dynamic is that people had long used Rich’s statements in various public fora as a proxy for an official “roadmap”/status.  Thus, when Rich isn’t around to comment on direction, people start wondering about the fate of the entire endeavor.  Without the sort of public declarations/monitors of activity described in the prior section, this sort of speculation is probably unavoidable.

Raw Data

You can get the raw data here.  If you are particularly interested in the Clojure community, lead a significant open source project, or write about, teach, or promote Clojure, you would be well-served to browse around the raw results (especially through the free-form comment fields and what people noted for their “other” choices on certain questions).  Tip: don’t bother trying to read the long-form responses in the Google Docs UI, which seems to handle scrolling of spreadsheet rows containing lots of content very poorly — export to Excel or something.

Final Thoughts

Putting together this survey has again been fun, informative, and a good reminder of just how vibrant and promising the Clojure landscape is.  There’s been a lot of good news and good progress over the past year, and it’s clear that this has been accompanied by a lot of good growth, too.

The coming year will see the second Clojure Conj, the release of Clojure v1.3.0, and who knows what other advancements and surprises.  If you’re a Clojure programmer, I hope that reading this has inspired you to jump in to find a corner of the Clojure landscape that you can help grow and tend.

Clojure offers a number of different forms that define types (and generate Java classes). Choosing between deftype, defrecord, reify, proxy, and gen-class can be a tripping point for those new to Clojure.

I’d obviously like to make such decisions easier for everyone.  I know that many people learn best through visual aids, so I’ve been working on a flowchart that attempts to encapsulate some of the significant choices that go into deciding between the different type-definition forms.  A draft of it is below; let me know if it is helpful to you (or not!), how you think it could be made better, etc.  The most recent version of the flowchart, the “source” OmniGraffle file, and many translations of it (feel free to contribute another one!) will always be available here.

“The ‘Interop Zone’” demarcates use cases (e.g. needing to define multiple constructors) and forms (proxy and gen-class) that are exclusively the domain of Clojure’s JVM interoperability support. Using these forms is fine, but be aware that by doing so, you are stepping outside of Clojure’s “native” abstractions; unless you are defining a type for the express purpose of meeting interoperability requirements, one of Clojure’s simpler type-definition forms may suit your needs better.

Updated 2011-07-05 18:28 UTC based on initial feedback. Thanks!

Updated 2011-07-07 17:38 UTC — added “The ‘Interop Zone’”

Updated 2011-08-09 16:30 UTC — Japanese translation provided by OGINO Masanori

Updated 2011-08-23 17:33 UTC — Portuguese translation provided by Paulo Suzart

Final update: The most recent version of the flowchart, the “source” OmniGraffle file, and many translations of it (feel free to contribute another one!) will always be available here.

I’ve finished putting together the State of Clojure 2011 survey, which is embedded below.  It will remain open for input until Monday, June 20th.

As was the case last year, we are likely on the verge of a new “major” release of Clojure (v1.3.0 this time around), and the apparent state of affairs can be described similarly:

The Clojure community is larger than it ever has been, and shows no sign of slackening its growth.  It seems like now would be a good time to take stock of where the community is, how people came to use Clojure, and how it’s being used in the world.

Hopefully enough responses will come through that we’ll be able to get a good picture of the current state of affairs, and maybe a little insight into where Clojure can and should make headway in the future.

As with the State of Clojure 2010 survey, I will post again sometime shortly after the survey closes with all of the captured data, some pretty charts, and whatever attempts at witty, (un?)biased commentary I can come up with. ;-)  Seriously though, we’ll have prior data available this time around, so some interesting trends and comparisons might be inferred.

The survey has now ended.

A summary of results and all the raw data are now available here.