Buy a signed copy of ‘Clojure Programming’, help the EFF!

Through a strange set of circumstances, I recently came into a decent-sized cache of copies of Clojure Programming (which I co-authored, in case the blatant self-promotion to the right didn’t tip you off already).  The only other likely option for them was to lay disused for some years, and then end up in the bin; so, I bought them (at a steep discount) from the prior owner, not quite knowing what I’d do with them.

If I didn’t live in the hinterlands, I’d just drop them off at the nearest gathering of Clojure enthusiasts; but, lugging boxes of books hours away to Boston or New York had little appeal.  My next thought was to simply sell them, but the economics are just crummy, given the PITA logistics of handling payments, boxing, and shipping for one book at a time, all for something like $15 or so net. (Yes, sorry, I’ll file this under #firstworldproblems.)

But, a better idea came eventually, one that I hope enough of you will cotton to to make an impact:

Buy a signed copy of Clojure Programming; all proceeds go to the Electronic Frontier Foundation

Surely you’re familiar with the EFF already; if not, you should be.

The “rules” here are very simple; basically, this is a first-price sealed-bid auction:

  1. Send me an email indicating how much you would like to donate to the EFF, $100 minimum, exclusive of the shipping necessary to get the book to you.
  2. The top 5 pledges received before March 5th, 2014 will receive a new copy of Clojure Programming, signed by me (FWIW, etc).  I’ll write whatever else you’d like in terms of a salutation, personal message, etc.
  3. The cumulative amount pledged will be donated to the EFF, in care of those that made those top 5 pledges.

If it’s not completely clear, I’ll not be keeping a single cent of the pledged amounts; everything will be going to the EFF.  In fact, I’d like to use Dwolla if possible for all donations, which would effectively eliminate transaction fees; I’m pretty sure the EFF can do more with that 3% than Visa, Mastercard, and Paypal.

Sound good?  Pledge away, and feel free to ask any questions here or via Twitter.

Posted in Clojure, Clojure Programming (book) | Tagged , | Leave a comment

Results of the 2013 State of Clojure & ClojureScript survey

Two weeks ago, I opened up this year’s Clojure & ClojureScript survey.  Now, you get to see what everyone said, and we all get to speculate on what it means.

First, some process details:

  • The survey was open for ~7 days, from Nov. 5th – Nov. 12th.
  • I announced the survey here, on my personal Twitter feed, and via a total of four messages to the main Clojure & ClojureScript mailing lists.
  • 1,061 responses were received.

One immediately-surprising thing is that fewer people participated than last year; not by a staggering amount, but enough to make me pause.  Of course, there’s a ton of uninteresting reasons why that might be the case: the time of year the survey was offered, the new collection method being used (PollDaddy vs. a Google Docs form last year), the size of the survey (23 questions vs. 13 last year), and the peer presence of ClojureScript in the survey (which accounts for the increase in question count).  Of course, all of this comes with the caveat that I’m far from a professional pollster, so there are tons of factors that I’m surely unaware of that would impact participation (maybe some that I’m responsible / to blame for).

(Late note: people have mentioned that, in contrast to prior surveys, this year’s did not get any significant placement on Hacker News et al.  That may further explain the drop-off in responses.)

Per-question tl;dr

What follows is a super-brief summary of the results, with my own opinions interspersed.

Do you use Clojure, ClojureScript, or both?

The community is effectively split between those that use only Clojure, and those that use Clojure and ClojureScript.  Only 7% of respondents use only ClojureScript.

To me, this implies that ClojureScript has not (yet?) attracted a population looking to use it, and it only (as opposed to other languages that target JavaScript, e.g. CoffeeScript, that are used in conjunction with a diverse set of backend languages / architectures as appropriate).  A rosy speculation might be that those coming to ClojureScript from other languages are so enamoured of the language and its model that they also adopt Clojure, but it’s probably more realistic to say that most ClojureScript usage is simply a result of Clojure developers wanting to target JavaScript environments with a minimum of language and data model pain.

In which domains are you applying Clojure and/or ClojureScript?

In a few words, web development, open source, math and data analysis, and building commercial services and products using databases.  There aren’t a lot of surprises here.

Notable is that ~20% of respondents are using Datomic in some capacity.

The relative distribution of the domains really hasn’t changed since I started running the survey.  At this point, the only utility of this question is to limit one’s view of the rest of the results in terms of what people are working on.

Order these aspects of Clojure/ClojureScript to reflect how useful each have been to you and your projects…

It’s no surprise that functional programming, REPLs, immutability, “ease of development” (an ill-defined aspect to include in the list, I now realize), and host interop are hugely impactful; these are core to Clojure’s rationale.

On the other end of the spectrum, metadata and newer features like reducers and tagged reader literals are comparatively not getting a lot of love.  This is actually a bit unfair, especially with regard to metadata, but the stack-ranking nature of the question means that some useful bits were guaranteed to be lingering at the bottom.

What is your *primary* Clojure/ClojureScript development environment?

This question changed from prior years (people could choose only one development environment this time around), so comparisons with prior years would be inappropriate.

  • Emacs is preferred by over half of respondents, and nearly all of them are using cider/nrepl.el now; effectively no one is using SLIME or slimv anymore.
  • Likewise, vim users have largely moved over to vim-fireplace.
  • Light Table is the third most preferred development environment, just beating out Counterclockwise.
  • In a very short period of time, Cursive Clojure has overtaken the usage of the La Clojure IntelliJ plugin.

There are now a number of Clojure/ClojureScript implementations targeting runtimes aside from the JVM and JavaScript. To what degree are you aware of or using each of these implementations?

Very few people are using these alternative implementations, though up to 20% of respondents are evaluating at least one; the “winner” there is clojurec, which makes sense given some of the most popular complaints around Clojure, i.e. its general unsuitability for command-line utility development and the difficulty of using native libraries from within the JVM.

Re: Clojure

How would you characterize your use of Clojure today?

More than half of respondents are using Clojure at work, which is a big jump compared to the last two years, when only a third of respondents were so lucky.  This is nothing but good.

What version of the JRE/JDK do you target?

We are generally early adopters, and that carries over to JDK selection: 75% of respondents are using 1.7, 5% are on some pre-release of JDK 1.8.  JDK 5 is dead now, and JDK 6 is on its way.

What tools do you use to compile/package/deploy/release your Clojure projects?

This is another question where I changed the survey to force people to make one selection, so comparisons across years don’t make sense.  Leiningen dominates the Clojure project management / build space.

Name *one* language feature you would like to see added to Clojure.

This was a free-form question, and I’m not going to attempt to summarize the responses. I’d encourage everyone to click through to the full survey results and look at some of these yourselves (maybe filtered to focus on the “types” of Clojure programmers / usage you are interested in).

Update: Alex Miller has summarized this question’s responses into a ranked set of categories of features.  This is a great way to get an overall impression of what’s top-of-mind among participants without weeding through the raw responses yourself.  Thanks, Alex!

In general, are the following statements true when applied to the Clojure libraries available within your domain(s)?

This question attempted to gauge general sentiment with regard to library quality on a couple of different criteria.  In short, Clojure libraries are easy to find, their maintainers are receptive to feedback and patches, they are technically of high quality, but they’re not always very well-documented.  None of that is surprising or particularly different from last year.

What has been most frustrating for you in your use of Clojure; or, what has kept you from using Clojure more than you do now?

Clojure is not generally suitable for building command-line tools, though people wish they could use it for that (perhaps ClojureScript and/or something like clojurec will help here over time).  As in prior years, people remain unsatisfied with documentation and development environments.

“Staffing concerns” is new as the #2 reported frustration, which I think is significant: I certainly know that companies using Clojure have a hard time filling openings lately.  Perhaps this is correlated with (maybe) increased commercial use of Clojure, given many more people using Clojure at work this year; it may also be a reflection of current/recent macroeconomics.

What do you think is Clojure’s most glaring weakness / blind spot / problem?

Again, you should go look at the raw results to see what people highlighted as problematic areas in Clojure.  I will call out this response though, as it was both at the top of the list of responses for this question, and made me chuckle:

“Where are the docs?”

“Read the (tests|docstrings).”

“Did you just tell me to go fuck myself?”

“I believe I did, Bob.”

(Apologies to @jrecursive.)

Re: ClojureScript

How long have you been using ClojureScript?

ClojureScript is very new, and in general, all ClojureScript programmers are new, too: effectively all of us have been at it for a year or less.

How would you characterize your use of ClojureScript today?

Despite ClojureScript’s youth, over 25% of its users are using it at work.  I find that amazing, and is a testament to the relative maturity of the language, i.e. it started off benefiting from the lessons that were learned in building Clojure and bringing it to the point that it’s at today.  Of course, that doesn’t yet reflect in the maturity of ClojureScript’s implementation.  Things there are getting better fast, and will continue to do so.

Everyone else is basically just tinkering, which makes total sense right now.

Which JavaScript environments do you target?

Browsers dominate here, which makes sense.  However, people target a bunch of other environments with ClojureScript too, from node.js to app containers to plv8 to Riak’s mapreduce.  Basically, if it can run JavaScript, I’m guessing someone is going to target it with ClojureScript.

The tricky thing about this is what this implies for those of us targeting more than one environment with our Clojure[Script]ing: while 95% of a codebase may be perfectly portable among all sorts of runtimes one can target from Clojure and ClojureScript, many of the options available for dealing with that last 5% are suboptimal, especially insofar as adding an additional level of indirection to represent runtime differences conflicts with the typical objective of that 5%, namely performance-sensitive bits that need to touch runtime-specific / interop facilities.  Fixing / working around this has been a defcon 3 yak for me for some time, thus cljx…maybe it’ll help you out if you’re contending with the same issues.

Which tools do you use to compile/package/deploy/release your ClojureScript projects?

Just like with Clojure, Leiningen (with lein-cljsbuild) is the overwhelming favourite for ClojureScript build/project management.  This is no surprise given huge overlap between those using ClojureScript and those using Clojure.

Which ClojureScript REPL do you use most often?

The results here are very surprising, and sad to me: more than a quarter of ClojureScript developers don’t use a REPL at all.  Looking at the responses around what can be improved about ClojureScript, the ease of use of existing REPL options is easily the #1 reported problem, and is clearly preventing people from using what is reported more generally as one of the most useful parts of Clojure/ClojureScript.  This is absolutely something that is hurting ClojureScript adoption, especially insofar as one of the first things people reach for when language tinkering is the REPL and ways to make it work with one’s other tools.

I’m doing what I can to help this problem by working on Austin, an alternative project- and browser-REPL implementation that aims to make ClojureScript REPL-ing as easy and as indispensable as it is in Clojure.  If you’ve had a hard time getting ClojureScript’s browser REPL going reliably, or would like to not depend on e.g. Rhino for project-connected REPLs, take a look; it’s far from done, but what’s there is easier to use and somewhat more flexible than the baseline.

Name *one* language feature you would like to see added to ClojureScript.

Just like with the Clojure analogue, this was a free-form question that I’m not going to comment on here. Click through to the full survey results and look at some of these yourselves.

What has been most frustrating for you in your use of ClojureScript; or, what has kept you from using ClojureScript more than you do now?

As mentioned before, difficulty using REPLs is the biggest deal here, followed by “difficulty debugging generated JavaScript” (something that I know has been a big focus of David Nolen & co. with source maps and such).  Other big problems include using JavaScript libraries (something I’m queued to improve), followed by the eternal frustration around documentation, and my personal favourite, “coping with JavaScript”.

That last one is not something we can do much about very quickly in the ClojureScript community, but it’s definitely one I share; having relied on the JVM and JDK for the last 15 years, it can sometimes be incredibly frustrating to have to work around the shortcomings of the JavaScript runtime environment, language problems, and holes and misfeatures in the standard library.  Hopefully we’ll be saved eventually by some potential improvements coming from the ECMAScript standardization/evolution process.  In the meantime, perhaps there are some clever things we can do to blunt the pain a bit.

What do you think is ClojureScript’s most glaring weakness / blind spot / problem?

Here is where you can leave any general comments or opinions you have…

The last two questions both collected free-form responses, only the first of which is ClojureScript-specific.  The second is always my favourite part of the survey, where people can write anything they want about the survey, our languages, or our community; it’s overwhelmingly positive, and reminds me why I’m proud to be part of all of this.  The only one I’ll quote that appears a bunch:

Thanks for the survey Chas!

My pleasure, thank you for participating!

Survey data access

Complete access to the survey reports (with a pretty snazzy filtering and drilldown UI) as well as all the raw data are available here; the password for that shared view is “HvmqkIS3dx3″.

Posted in Clojure | 2 Comments

2013 State of Clojure & ClojureScript Survey

This is the fourth year in which I’ve offered up a “State of Clojure” survey.  As before, my intention is to get a sense for the size and overall level of activity in the Clojure community, and to capture a certain zeitgeist of the community’s mood, practices, concerns, and so on.  If you’re interested in history, you can see the results of all prior surveys:

The big change this year is that questions related to ClojureScript are included such that it roughly shares the limelight with Clojure.  In part, that’s selfish; I started using and contributing to ClojureScript seriously over the last 18 months, so I’m interested to have some perspective on what other users care about.  More importantly, there’s no denying that ClojureScript has matured significantly over that period, and has attracted a great deal more attention. Especially given its potential, it’s time it took its proper place in the survey as a peer implementation of the Clojure Principles we’ve all come to appreciate.

Results from the 2013 State of Clojure & ClojureScript survey are here.

This year, I’ve moved to hosting the survey with PollDaddy, which has graciously set me up with a ‘pro’ account; this will allow us to get far more advanced analysis out of the results than were ever possible with the primitive Google Docs form I’d used up until now.

As before, I’ll follow up with the results, some charts and graphs, and some sad attempts at witty commentary.  Of course, all of the raw data will be available as well.

Finally, please do what you can to spread around this survey to those that you know of that are working with Clojure and/or ClojureScript.

Posted in Clojure | 6 Comments

My Mom has Multiple Sclerosis

She’s likely had it since at least ~1983; the first sign was a temporary bout of optic neuritis that left her half-blind for a month or so.  All was well for years, until 1996: we were walking out of the DMV — we went so I could get my driver’s permit — and she suddenly stopped walking in the parking lot…and couldn’t move any further.  After a minute or two of my asking what was wrong, and her looking a bit panicked and not knowing what to do, her legs started working again.  Turns out, I had gotten my driver’s permit just in time; I drove home that afternoon.

In the years since, a lot has happened: diagnoses (“secondary progressive multiple sclerosis”), various attempted treatments, the walker, then the wheelchair, then full quadriplegia.  My father, my wife, and a cadre of part-time nurses (thanks, everyone, for your contributions to Medicare/Medicaid/MassHealth) have helped her remain as independent as someone in her position can be.

No doubt, it’s been hard on everyone involved, but I think we’ve done okay, and made the most of a genuinely shitty situation.  Of course, it’s been hardest on my Mom.  I can’t imagine what it’d be like to be in her position (though believe me, I’ve tried), but — understandable dips in the road aside — she has remained remarkably upbeat, engaged, rational, and hilariously concerned with others’ well-being.  Courageous, that’s what she is, and I’m so proud of her for it.

But, I write all this not to share my perspective, or tell my story, but to suggest that you should listen to her tell hers: she recently started a blog, Living with Advanced Multiple Sclerosis.  If you’ve gotten this far in this post, you should go read it.

One thing that did decline as her disease progressed was her interest in her most fervent passion in life, literature and writing.  She was an English major in college, and along with helping me become the language pedant I am today, she helped cultivate my love of rhetoric and the written word.  While it was sad to see her stop reading and writing for so long (it can be hard to keep up with it when neither your arms or eyes work well, or at all), it made seeing her start to write again that much more joyful.

So, check out her blog, and pass it along to anyone that you think might appreciate it.  I know she’d love to know that other people are following along, especially if they are also on a journey with a degenerative neurological disease like Multiple Sclerosis, Parkinson’s, Muscular Dystrophy, and so on.  It’s no Shakespeare (something she’d readily admit; since her nurses type for her, she can’t meticulously edit and fret over every word for hours like she once did).  No, it’s better: her unvarnished voice talking about life, something I love seeing on the “page” again.

Posted in Uncategorized | 1 Comment

[ANN] @IMHO

Update: I’ve shut down IMHO. It was amusing while it lasted, but it’s clearly not a domain in which I have a fundamental interest. I hope you all enjoyed playing with it while it was among us!

I don’t write stupid Twitter apps often…but when I do, they’re really stupid.

— Me, wearing a smoking jacket

I’ve always enjoyed Twitter, but I’ve never built an app of any kind around it, or done anything with Twitter data.  Insofar as that’s roughly the modern equivalent of a ‘hello world’ program, I was perhaps lacking in some critical way.

Well, no longer.  I, too, have built a Stupid Twitter App™: go check out IMHO (or its companion Twitter handle, @IMHO)!

logo-med-darkWhile Twitter has many roles — some quite important — everyone uses it as a dumping ground for their opinions.  IMHO (a common Internet colloquialism meaning “in my humble opinion”) will maybe provide an entertaining and perhaps informative view of those opinions, in aggregate.  I’ve seeded the site with just short of 2 million opinions culled from an archive of 90 million tweets; but, from here on out, new opinions will only be added if they are tweeted at @IMHO, like so:

.@IMHO The Celtics will get revenge next year!

Through a combination of some crude natural language processing and a lot of hard-working squirrels, opinions are parsed and indexed by their subject/topic.  Oh, but leave your nuanced stances at the door: only simpler, discrete assertions will be recognized.  Madness?  No, this is Twitter!

Right now, only reverse-chronological listings of opinions are available (far more interesting visualizations are of course close at hand); perhaps I’m just that puerile, but I’ve found even that simplistic view entertaining enough for now.  Check out some popular/contentious topics:

There’s plenty of snort-worthy gems in there, if you care to go fishing.  Again, some better UI will make it easier to surface them, which will be added if the service catches on in any way.  So, if you have an opinion you’re going to put on Twitter, put a .@IMHO on it.

Posted in Announcements | 1 Comment

100% time

Perhaps you’ve heard of “20% time”. In many ways, it or something like it are table stakes for many software folk, and perhaps other creative specialists as well; i.e. if a company doesn’t offer something like 20% time, it may have a hard time attracting top talent, and could end up suffering by not profiting from the results of the outside-the-box sorts of ideas and work that emerge from people’s 20% time.  Some organizations — Valve comes to mind as the most prominent — even make it policy that staff are to use the “law of two feet” to self-organize, with the theory that more impactful work will emerge from the resulting economics.

I relate to this insofar as I’ve been lucky to have wandered into having what I call, tongue-in-cheek, “100% time”.  Most discussions of 20% time seem to characterize the mix as being 80% slog, 20% “freedom”. In contrast, “100% time” is a mix of near-complete professional and personal freedom where I can be available to every opportunity that comes my way.  Whether related to new business, following new creative inspirations (in programming, web or graphic design, or writing), pursuing scholarly interests, keeping myself healthy, traveling at length, enjoying the company of and taking care of family, volunteering for causes I believe in, or slacking to recharge, 100% time means that I choose what to care about, and then dedicate all my energy to making that choice have impact.  Having had the opportunity to live like this and becoming acutely aware of it, I’m nearly certain I won’t be able to “go back” without a fight.

I don’t write this to brag.  More than anything else, if 100% time seems out of your reach, I hope to be some proof that it’s not.

There’s nothing in my past to suggest that I should be where I am, doing what I am doing: no family money, no name-brand school (or diploma, for that matter), no powerful connections.  In fact, I came very, very close to getting stuck in a “regular” job ten or eleven years ago, after sinking $140K in debt trying to start the first incarnation of Snowtide.  I thought that had been my one “shot”: after failing dismally and being forced to take any work I could get (initially landing in a hyper-dysfunctional office run by maniacal Russians…), I figured that my life’s trajectory was fixed.  I know I shouldn’t lament a “professional” career with stable companies — it was far more than I had any right to expect — but, perhaps irrationally, I wanted to be king, with direct control over my life and my future.

Thankfully, I’m either too stubborn or too stupid to give up.  I extracted PDFTextStream from the smoldering ashes of my wayward startup, and managed to build a small but reliable business serving fabulous customers in a technically-challenging niche.  Somewhere along the way, I discovered that the biggest benefit of entrepreneurship was not money (as many have said, far safer routes to much larger piles of cash exist elsewhere; go be a banker or management consultant if that’s where your objectives lie), but time, and the freedom that comes with it.  Once my livelihood and income were decoupled from the time I had to dedicate to earn it, I felt like I finally understood the concept of opportunity cost and the aphorism of spending time: you will exist for only a finite duration, and you’d best ensure that that precious capital is used wisely to build the most value possible.

How you personally define “value” is where all the fun and challenge comes from.  Build bigger, better, more beautiful things; learn to make music and art and drama; inspire an empire and then go save the world; love friends and family and neighbors and strangers.  Do what you can to have the opportunity to make those choices yourself, so you can be the best person you can be, and make the most of the time you’ve been allotted.

Posted in Entrepreneurship | 12 Comments

Cargo-culting to success and understanding

In doing my part to further the fortune-cookie bullshit cycle that is Twitter, I tossed out this nugget yesterday:

Little did I know that it would spur such conversation, debate, DMs, emails, and so on.  I was going to expound on the original tweet in eight parts, but thought better of it.

Cargo culting is an actual thing, of course.  Thanks to Justin Sheehy for this f’rinstince:

He rightly pointed out that real cargo culting is all about mimicking the “trappings of others, not their essential behaviour”.

Mimicking triviality certainly happens — witness the fads that drift in and out of tech culture around the most trivial things, e.g. the super-minimal desk/workspace, actually using a hammock for “hammock time”, the revolutions in agile/xp/lean/scrum/etc. nomenclature, and so on — but I don’t think that’s what most people mean by “cargo culting”, at least when it comes to software, programming, and related design.

By “cargo-culting”, I was generally referring to doing something without properly knowing why it might be good or bad.  In most cases, doing this is a recipe for disaster, especially if the person doing it is of the incurious sort that is looking for permanent shortcuts.  “Keep adding indexes if your query is slow”, “go look in the ‘pattern’ book when trying to design something”, “use Mongo if you need to be web-scale”.

(The dangerous bit is that we all occasionally cargo-cult things, implicitly, simply because we are social creatures, and we’re inevitably influenced by certain patterns and design philosophies and technical approaches being baked into the fabric of our time and place in the world.  My sense is that many discontinuous innovations can be strongly correlated with someone becoming explicitly aware of these undercurrents, rationally re-evaluating their bases, and offering an alternative better suited to modern times and uses.)

What I was tweeting about is a different thing, though.

Especially when I’m groping into a new domain, I often ape others’ work with abandon, and with only a dim view of the ‘why’ of the motivating design problems and decisions.  Doing so produces technical, design, and understanding debt, but I take it on willingly.  Making constant progress is often more important and more useful to me to than methodically building a formal understanding of the theory or practice related to my current task.  As I go along in the work, I continually look for ways to understand those decisions I previously adopted wholesale.  Because of the bias towards constant progress, I generally have the benefit of having a working system I built in front of me, so I have a tangible sense of the impact of those decisions.  I then can carry on having understood the original ‘why’ motivating them; and, if I’m feeling adventurous, I can use that understanding to usefully re-evaluate those decisions, and maybe make different ones to yield a better result.

Maybe I’m being too loose with the terminology, but the first part of this process certainly sounds like “cargo-culting” in software to me.  The difference is that:

  1. I explicitly acknowledge that I’m taking a shortcut, with the distinct intention of returning to the topic later.  (Not doing so would be an abject failure on my part.)  This is the “first approximation” part of the notion: the shortcut is a bootstrapping mechanism, not a final destination.
  2. I am extremely selective when it comes to whose work I’ll choose to look at seriously.  Code pasted into a Stack Overflow answer doesn’t qualify, nor does whatever is found in most popular “technical” books.  Libraries and systems built by people who have spent decades, careers working on problems similar to those I’m facing? Getting closer; but even given that sort of “population”, care must be taken to match up the matrix of requirements as closely as possible.  e.g. if I’m in the neighborhood of distributed systems, taking hints from someone focused on embedded applications may be fraught with peril.

I’ve never been able to read a dissertation or book or three, and *foom*, produce out of whole cloth something well-designed, efficient, and extensible — but I’ve seen others do just that.  So, I know that what I’ve discussed here is an inefficient path, at least in certain domains and for certain people.  I think it is a natural response to attempting to build nontrivial things given fifty-ish years of software and engineering history to consider.

Finally, the terminology.  Perhaps given this notion of historical artifact, a better phrase than “cargo culting” might be “anthropological reconstructive software archaeology”?  Doesn’t quite have the same ring to it though, eh?

Posted in Craftsmanship | 7 Comments

Mostly Lazy, back in the saddle

I was stoked to reboot Mostly Lazy by talking yesterday with Chris Houser (a.k.a. Chouser), this time via Skype.  It’s good to be back, so go check out the latest episode, maybe follow the @MostlyLazy twitter feed (who knows, it might not suck), and suggest some future topics, questions, etc.

Posted in Clojure | Leave a comment

Results of the 2012 State of Clojure survey

A few weeks ago, I opened the 2012 State of Clojure surveyPer usual, I wanted to take the Clojure community’s collective temperature, see where we all came from, what we’re using Clojure for, what weaknesses and problems exist, and hopefully provide some results that would allow everyone to calibrate priorities and such.  This was the third such survey I’ve conducted, and it was a little different than prior efforts:

  • Now that there are a bunch of rapidly-maturing alternative Clojure implementations targeting non-JVM platforms, I wanted to see how those alternatives were faring in terms of uptake and usage.  So, the survey asked participants to characterize their usage of each of the Clojure implementations that I knew of that have matured to some minimal level.
  • A FAQ among language and library implementers has always been, “Which version of the JVM should I target?” I can’t say why I didn’t include a question relevant to that topic from the beginning, but it was included in this one — and the answers were surprising, at least to me.  A similar question was included that deals with which environment(s) people are targeting with ClojureScript.
  • As Clojure is used more widely and it becomes more and more clear that the language itself is stable, effective, and reliable, the next most pressing concern is the state of Clojure libraries.  Survey participants were asked to grade Clojure libraries, in general, on a number of different scores, from effectiveness to ease of discovery to documentation.

I hope you find the results interesting and useful.

Vitals

First, some facts about the survey and how it was conducted. It was available for approximately 7 days (Wednesday night through Monday afternoon), during which time results were not available.  The survey was announced primarily via Twitter and two messages to the main Clojure mailing list, which has approximately 6,700 subscribers. 1,372 responses were receiveda surprising 2x increase in participation over last year’s 672 responses. As I’ve said before, I’m not a statistician, nor am I a professional when it comes to surveying, polling, or data collection of this sort in general, but this would seem to be a very fair sampling of the Clojure community.

(I’m not sure what to make of the huge increase in participation.  I don’t think there’s any question that more people are using Clojure now than there were last year, but a 2x increase seems overoptimistic, especially given my bias.)

Now, on to the survey results.  See the links at the end to the raw data to satisfy all your statistical urges.

(Note: Any question with results that sum to more than 100% allowed respondents to select more than one response.)

Q: How long have you been using Clojure?

How long have you been using Clojure?

Good things to see: more new people trying Clojure, and those that were around last year and the year before sticking around.

New this year is the addition of the “I’ve stopped using Clojure” option.  Now that Clojure’s been around for long enough that some may have used it, and eventually stopped using it, I thought there might be lessons to be learned in what prompted such decisions.  I was surprised that only ten participants indicated that they’d stopped using Clojure; of course, people that have checked out are less likely to fill out a survey, but I was expecting more than ten nonetheless.  Only three of those left explanatory comments, so there’s hardly anything to generalize over.  But, there is one comment I’d like to highlight:

Clojure’s greatest strength is also it’s greatest barrier to acceptance into the mainstream corporate IT world – namely, it’s LISP syntax (which I personally like quite a bit).  Scala appears to be easier to pitch to corporate IT.  Thus, (for now) I’ve been working on pitching functional programming via Scala to our IT department.  The LISP syntax is to difficult a promote in a world dominated by Java and .Net.

Ah, the overwhelming power and ease of curly braces and monadic contravariant type classes? ;-)  (That phrase may be meaningless; my Scala vernacular is rusty these days.)  In the end, I don’t know that pitching Scala as a “better Java” because it can be made to look like Java is doing anyone any favors; as far as I can tell, idiomatic Scala is as different from Java as Clojure is from Java, in both syntax and semantics.

If you’re involved in corporate IT environments and have an interest in Clojure, you would do well to take in Neal Ford’s master plan for Clojure enterprise mindshare domination from last year’s Conj if you haven’t yet.  There’s no silver bullet therein, but I think it’s quite reasonable to expect “engaged” development organizations to be receptive to anything that will allow them to raise their game, regardless of parens vs. braces or similar silliness.

Q: What language did you use just prior to adopting Clojure — or, if Clojure is not your primary language now, what is that primary language?

Per usual, the majority of Clojure programmers “come from” Java, Ruby, and Python.  This is the third year of identical results, so this question will likely not return.

Q: If Clojure disappeared tomorrow, what language(s) might you use as a “replacement”?

Again, nearly identical to prior years’ results: relative to where people came from prior to using Clojure, functional programming (Haskell, Scala, Erlang) and lisps (Scheme, Common Lisp) are preferred.  Also mirroring prior years’ results, a number of people left “Other” responses like:

too depressing to contemplate

I agree.

Q: How would you characterize your use of Clojure today?

How would you characterize your use of Clojure *today*?

The proportion of people reporting at-work usage of Clojure is up again, 3 points higher than last year, and 11 points higher than in 2010. As important and promising as this is, I’m actually more heartened by the fact that the proportions of people that are using Clojure for hobby projects has remained constant: I think it would be a great loss if Clojure ever became one of those languages that is used at work, but shunned when one wants to do some hacking for fun.  That capacity for enabling play is a huge source of the creativity and liveliness that makes using Clojure not only sensible, but satisfying.

Q: In which domain(s) are you using Clojure?

In which domain(s) are you using Clojure?

This chart is identical to last year’s results.  Web development and math/data analysis remain ubiquitous and lots of people are contributing to open source Clojure projects.

Most of the “Other” domains are scientific in nature, interestingly enough:

  • scientific computing
  • biological research
  • bioinformatics
  • healthcare
  • medical (mass spec) data analysis

…and so on.  There’s also a solid chunk of people doing AI, NLP, and semantic web stuff.

Q: While Clojure started as a JVM-only language, there are now multiple implementations targeting different runtimes and environments. To what degree are you using each of these implementations?

The last year has seen an explosion in the number of runtimes targeted by various Clojure ports, reimplementations, and adaptations.  This question was answered via a grid of options, but a better visualization than a set of typical bar charts is not ready at hand; so, first the charts, and then a couple of words:

To what degree are you using Clojure?

To what degree are you using Clojure.CLR?

To what degree are you using ClojureScript?

To what degree are you using clojure-py?

To what degree are you using clojure-scheme?

To what degree are you using clojurec?

ClojureScript clearly has the most traction and attention of all of the alternative implementations — nearly 20% of respondents are using it in some non-investigatory capacity — but clojure-py, clojure-scheme, and clojurec are also each being evaluated by nontrivial numbers of programmers, despite their relative youth.  Targeting JavaScript, Python, Scheme, and C (and thus to all sorts of platforms, including iOS, embedded devices, and so on) have long been of interest to Clojure programmers, so this immediate attraction is not surprising.

This is in contrast with ClojureCLR, and demonstrates how critical the targeted platform is to the success of a language implementation: despite ClojureCLR being complete and well-supported by the tireless David Miller, 70% of respondents just don’t have any interest in it, and only 17 respondents are using it in any capacity.  My best guess at explaining this is that those targeting .NET are fundamentally unwilling to consider languages not officially supported by Microsoft…witness the stasis of efforts like IronRuby and IronPython compared to JRuby and Jython and the other dozens of stable, active languages on the JVM.

Aside from any technical matters, this dynamic alone is enough to justify Rich Hickey’s prehistoric decision to focus Clojure on the JVM rather than the .NET CLR, and should be a factor for anyone considering working on or adopting alternative Clojure implementations: the target platform cannot be a footnote or secondary consideration.

Finally: a couple of comments elsewhere either pined for a Clojure implementation that targeted Lua, or helpfully pointed out the existence of clojurescript-lua.  Clojure / Lua fans, check it out! (If a couple someones let me know it’s up to snuff, I’ll definitely include it next year.)

Q: If you are using Clojure on the JVM, which JRE/JDK version do you target?

Which JRE/JDK do you target?

Only 5 respondents indicated that they target Java 1.5.  I honestly expected that number to be much, much higher.  I concede that there are surely plenty of Clojure programmers who aren’t floating around on the mailing list and twitter and who aren’t likely to participate in surveys — and such programmers might be more likely to be targeting older JVMs.  But, all caveats aside, this is fantastic news.

This means that, in general, library authors can safely target Java 1.6.  And, if it yields any benefits, Clojure moving to require Java 1.6 would appear to have few downsides (apologies to the few still targeting 1.5).

Q: If you are using ClojureScript, which environments do you target?

Which JRE/JDK do you target?

What surprised me here was that there’s a bunch of people using ClojureScript to write node.js apps!  That surely exposes my personal biases.

Notable “Other” selections here include Android and iOS (presumably through embedded web views?), as well as usage of various mobile-deployment toolkits like PhoneGap, Cordova, and so on.

In hindsight, I probably should have added “Mobile devices” and “Databases” (e.g. CouchDB, PostgreSQL) to the available options.

Q: How true are the following statements when applied to the Clojure libraries available within your domain(s)?

The JVM and other targeted platforms are known, stable quantities; and, especially with the release of Clojure 1.3 and 1.4 in the last year and the ongoing maturation of ClojureScript, the language is stable as well.  This leaves libraries as a certain unending “last frontier” of usability, in terms of programmers being able to get things done using Clojure and Co.  So, the survey asked if Clojure libraries, in general, are aptly described by a series of statements.

First, the charts, and their corresponding statements:

“They implement core functionality well.”

“They are more effective than analogous libraries in other languages.”

“They are easy to find.”

“Their maintainers are receptive to feedback, patches, etc.”

“They are accurately and adequately documented.”

Sweeping conclusions made from this data are probably suspect, but I’ll make a stab at it:

Clojure libraries are generally of high quality, and more effective than libraries people have used in other languages.  Further, library maintainers are generally collaborative and easy to work with.

On the other hand, the right Clojure library isn’t always easy to find, and many libraries are poorly or inaccurately documented.

There have been a couple of stabs at easing discovery, but not yet anything that’s caught fire (or, been regularly maintained, unfortunately).  At the very least, everyone should keep search.maven.org bookmarked for those libraries that are deployed to Maven central.  But, in the end, it may be that there’s no solution to this problem, insofar as it’s damn hard to track distributed development efforts with a centrally-planned solution; just look at how essentially rudimentary search.maven.org is, and that’s servicing one of the largest software development ecosystems in the world.  It may be that time and internalized community knowledge (e.g. “What is the best library for X?” “Y, of course!”) are the only things we’ll be able to count on.

(Those shouting “CPAN, CPAN!” at their screens now should really calm down.)

The documentation issue is, as we’ve seen in previous surveys and as we’ll see again later here, the most common thread of complaint amongst Clojure programmers.  More on this in a bit.

In hindsight, I should have offered only two options (Yes/No), which would have pushed people off the fence of “Some are”, and thus made it easier to get a clear picture of the state of things.

Q: What have been the biggest wins for you in using Clojure?

Compared to last year, mentions of protocols, records, and types went up by 50%, probably due to people starting to grok how to best utilize them.  The proportion that called out multimethods doubled, which I’m even happier to see.  Beyond that, functional programming, JVM interop, and the joys of the REPL continue to be people’s favorite benefits of Clojure.

Notably, very, very few people reported that the extensible reader introduced in Clojure 1.4.0 was a big win.  This is understandable given how new 1.4.0 is (released just four months ago), and so it may take some time for people to learn how and when to use tagged literals well.

Q: Which development environment(s) do you use to work with Clojure?

There’s been some movement in the area of development environments:

  • Emacs usage dropped 10%
  • Eclipse + Counterclockwise gained 4%
  • Usage of “Command-line REPL”s gained 5%

…but, I think most of the shifts from last year are noise.  What isn’t noise is the total collapse of usage of NetBeans + Enclojure; it was used by 13% of respondents in 2010, and now is used by just 1%.

Popular mentions in the “Other” column include:

I’ll talk a bit more about development environments later.

Q: Which toolchain(s) do you use?

Everyone uses Leiningen.  The other 5% just wish they could.  ;-)

There’s no data from previous years to compare this to, but whatever the curve of growth of Leiningen, technomancy & co. deserve a ton of credit for shaving the yaks and herding the cats to build it up into what it has become today.  People’s everyday Clojure experience is greatly enhanced by the get-out-of-the-way approach taken by Leiningen, and I think we’re all better off for it.

Q: Describe how you have participated in the development of the Clojure implementation or “contrib” libraries.

This is a bit of inside baseball, but I was curious to see how widespread participation is in the development of Clojure itself and the surrounding “contrib” libraries, and what might be standing in the way of people that might otherwise participate.

A couple of things jump out at me:

66% of respondents haven’t had a need to contribute to Clojure, to the point of not even needing to report bugs.  I think that’s a damn good number of people that have presumably had a distinctly positive experience re: language quality and completeness.

There’s rarely a week (or, day, perhaps?) that goes by on Twitter or in irc without some griping about Clojure’s contributions process and policies. However, just 85 respondents (~6%) cite that process or the Clojure contributor agreement (the most contentious part of those policies) as barriers to their contributing to the language and surrounding libraries; this makes me think that these issues are the favorites of a loud minority more than anything else.  On the other hand, 85 respondents represent 28% of the group that have the desire and need to contribute (i.e. excluding those that “haven’t needed” to contribute and those that are “unsure of their ability” to contribute).  It does seem worrisome that more than a quarter of potential contributors are opting out at least some of the time for reasons orthogonal to the code and documentation and work involved.

Q: What has been most frustrating for you in your use of Clojure; or, what has kept you from using Clojure more than you do now?

A lot of the most common and most frustrating problems were expanded upon in the free-form responses, so I talk about them a bit there (see below). Otherwise, just a couple thoughts:

The Clojure community remains a nice place to be, with only 2% of respondents indicating that they’ve had unpleasant interactions with others, the same proportion as last year.

I didn’t think it possible, but an even larger proportion of respondents than last year — now up to a third — indicate that documentation is a key problem.  This ties into the feedback seen earlier that library documentation is generally not what it should be.  To a large degree, this is a self-inflicted wound: both Clojure and its libraries are technologically sufficient and effective and useful, but many, many people are tripping on their way towards using them.

Given that the only other problems within 10 percentage points of the documentation issue are largely out of everyone’s direct control (issues using Clojure for scripting, likely hampered by the JVM’s startup time; and future staffing concerns, more of a communications and psychological issue than a technical or content problem), the best thing anyone can do to help Clojure succeed is to help make documentation and tutorials better for every skill level and domain, anywhere. I do what I can in my projects and elsewhere; please do what you can, too.

Free-form topics

Per usual, I included two free-form fields where people could write whatever they like: one dedicated to the biggest problems people think Clojure has (which allowed people to point out topics I didn’t offer as options in the prior question, or to further emphasize their choices there), the other for “general comments”.  Responses from their fields are included in the raw data linked below, as well as in separate HTML files for easier reading (spreadsheets are really poor for reading long-form text).  I encourage you to take a look, as I only provide a selective, slanted, biased, self-interested, unscientific overview here…

“What do you think is Clojure’s most glaring weakness / blind spot / problem?”

Predictably, the topics named in the responses to this question ranged all over the place.  Here’s a selection of common complaints, or tidbits I found interesting.

Documentation & getting started

I already flogged this above, but it’s worth noting the level of “enthusiasm” (a.k.a. “vitriol”) many people have with regard to the documentation for and around Clojure.  This is easily the most common topic in the “weak spot” comments.  There are few areas that are immune from such criticism, but much of it inevitably lands on clojure.org, its (lack of) connection to the development wiki (flawed as it is), and so on.

Specifically, there are a number of comments along the lines of:

http://clojure.org/getting_started should be replaced with https://github.com/technomancy/leiningen/blob/master/README.md

There’s been a lot of griping over the years about the “getting started” section of clojure.org, insofar as it leads newcomers through bootstrapping Clojure through a direct java -cp ... invocation on the command line, leaving them with the default, not-particularly-nice (sorry) REPL.  I’ve previously thought that the neutral stance taken on clojure.org is reasonable, insofar as a language shouldn’t be dictating (explicitly or otherwise) the use of a particular toolchain.  However, given the clearly universal acceptance of Leiningen into people’s hearts, and relatively high degree of difficulty that newcomers clearly feel with regard to the official first steps, I think it may be time to revisit that neutrality.  Leiningen, especially its v2.x incarnation, makes Clojure look and feel good “right out of the box”; since so many of the difficulties people report are around the immediate getting-started process, this simple recommendation could alleviate a lot of pain people are having.

Development environments

Many remain unhappy with the available development environments, and many express frustration with Emacs and the learning curve it entails (this not necessarily being an indictment of Emacs; if another editor were the most popular, it might receive the majority of scorn as well). However, I’ll go out on a limb and say that the number and strength of complaints on this topic were down somewhat from last year, which would correspond well with the 8% drop in reports of editors and IDEs being a leading frustration in the previous question compared to 2011.

Along these lines, a number of respondents wrote something similar to:

…every time counter clockwise gets another release, the world gets better.

I wholeheartedly agree.  Laurent has made a ton of progress with Counterclockwise in the past year, with Leiningen 2 integration that’s never failed me.  By all means check out the latest betas for absolutely excellent, dead easy-to-use Clojure code completion.  If you’re not happy with what you use today, give Counterclockwise a shot. (Disclaimer: I am — though not lately :-( — a Counterclockwise contributor.)

And, thanks to Relevance as well, for their ongoing sponsorship of Laurent’s work.

Hubris

I found this complaint interesting:

The hubris it’s champions have about functional programming.

I might be a good prototype of this commenter’s claim: I champion Clojure, am convinced that functional programming is superior to nearly all alternatives for most applications that I care about and that I see friends and peers building, and I share that conviction broadly.  Perhaps “hubristic Clojure goon” is the new “smug lisp weenie”?

The notion of “hubris” — excessive pride and arrogance — is interesting though. Should we scrape along with caveats and disclaimers, giving equal time to procedural programming and object oriented programming and the power of pointer arithmetic in order to prove our humility? One of the great things about the Clojure community in general is that it expects people to be adults capable of independent and self-motivated thought.  So, if someone — anyone — says something confidently, you should be happy and proud to disagree as long as you have good reason to do so.  There is nothing more cherished among Clojure programmers than well-reasoned discussion, and admitting error, mistake, or fault isn’t a taboo sign of weakness.

To me, this sounds like vitality and honesty, not hubris.

Development process snark

Whatever its merits or demerits, people have grown accustomed to laissez-faire collaboration styles (exemplified currently by the typical Github project), so comments like this:

contrib doesn’t seem very contrib-y.

and

Clojure/core’s apparent unfriendliness to community contributions.

aren’t uncommon, and underline the sentiment some expressed in the contribution-related question earlier.  Fair or not, there is definitely an undercurrent of frustration among some.

Android pain

An example comment:

One would expect clojure to have a better Android story.

Many people lamented Clojure’s apparent difficulty in being used to program Android apps.  I confess I’ve never tried myself, but I know that some people have gone to great lengths to build and publish Android apps written in Clojure, apparently needing to produce custom builds of the language in order to work around particularities in the Dalvik VM (stack size, JIT limitations, and classloader idiosyncrasies being some of the problems I’m vaguely aware of).  Making Clojure work well on Android is an effort being tracked, but one that I don’t really know the status of.

It’s surprising how many Android-related complaints showed up, even though only 3% of respondents reported using Clojure for mobile development earlier.  Perhaps that figure is so low because Clojure’s Android support is apparently just not up to snuff (and thus driving people to build apps using ClojureScript + e.g. Phonegap, or punting entirely and using Java or Scala or …)?

“General comments?”

Thankfully, the “general comments” field immediately followed the dedicated bitch-box ;-), so nearly all of the responses here were filled with whimsy and joy about how great Clojure is, e.g.:

Thanks to RH.

It’s a lovely language.

I love it. Programming became fun again like it was 15 years ago.

Missing question: “Do you like the direction Clojure’s going in?”  - Yes! :)

Clojurescript is the shit

<3

Clojure put the fun back in software development for me at a point where I was disillusioned and close to a career change (even though I started programming when I was 10yrs old). Now I’m programming Clojure 12hrs a day for the tech startup I founded and I absolutely love it.

…and, my personal favourite:

I love Clojure so much, I want to give it a kitten.

Me too! So, Clojure, have a kitten; her name is Mittens:

Mittens!

She’s seen enough Clojure at this point that she should be able to get a job writing the stuff by now…

Raw Data

All of the data gathered by the survey form are available:

If you are particularly interested in the Clojure community, are involved in Clojure projects, or write about, teach, or promote Clojure, you would be well-served to browse around the survey data to draw your own conclusions.


I’ll wrap this up with one final thought that came while combing through the survey data and compiling this post: all while Clojure and its community has stayed healthy and continued growing like a weed, and while ambitious hackers have helped the language bloom into a family of languages targeting different platforms, no fatal risk has yet shown itself.  Five years on, there’s been no crucial flaw found in the principled underpinnings of the language, and no poisonous dynamic has twisted its way into the community.  While problems do exist, they are tractable, and many will yield through time and goodwill, a commodity for which we thankfully do not want.

For anyone considering placing a bet on Clojure, and for those of us that already have, these are all very, very good signs.

Thus ends the third State of Clojure survey.  I hope you’ve found the above interesting, thought-provoking, and perhaps useful.

Posted in Clojure | 15 Comments

PDFTextStream now available free (as in beer)

PDFTextStream v2.6.0 was released today with a variety of small new features and a couple of bugfixes.  The bigger change is that PDFTextStream is now available free for use in single-threaded applications.

Because of the realities of the economics around developing and maintaining a product like PDFTextStream, its pricing has often been out of reach of many projects and very small organizations that really need high-quality PDF content extraction functionality.  That’s not to say that PDFTextStream is overpriced — it’s actually less expensive than other options — but that is small comfort to many that simply cannot afford or cannot justify the expenditure yet.

This change should fix that: if you have a smaller project, are working on a startup, are involved in information research, etc., you can now benefit from all that PDFTextStream has to offer.  And, if and when your architecture requires concurrent PDF processing, or your PDF content extraction workload is large enough to need to worry about properly utilizing your hardware and compute resources, you can easily upgrade to the unlimited, licensed “edition” of PDFTextStream to parallelize that workload.

It will be fun to see what people build now that PDFTextStream is gratisTry it out!

Posted in Announcements, PDFTextStream | Leave a comment