2012 State of Clojure survey

I’ve run “State of Clojure” surveys for each of the last two years (see results from 2010 and 2011), and the time has come for the 2012 edition.

The survey itself is embedded below.  It will remain open for input for approximately a week, until ~Thursday, July 26th.

A lot has happened over the past year: the first two non-Conj Clojure conferences (Clojure/West and EuroClojure) went off to great popular acclaim, Clojure 1.4.0 was released, more Clojure books have been published, and more and more people continue to be drawn into the language’s orbit.  The apparent vector of progress and activity seems to be approximately the same as it was when I wrote up the first State of Clojure survey:

The Clojure community is larger than it ever has been, and shows no sign of slackening its growth.  It seems like now would be a good time to take stock of where the community is, how people came to use Clojure, and how it’s being used in the world.

Hopefully enough responses will come through that we’ll be able to get a good picture of the current state of affairs, and maybe a little insight into where Clojure can and should make headway in the future.

As before, I will post again sometime shortly after the survey closes with all of the captured data, some pretty charts, and whatever attempts at witty, (un?)biased commentary I can come up with.  ;-) It would be great to see some follow-on analyses using the raw data: e.g., people doing game development are really unhappy with the state of libraries in their chosen domain; or, maybe people doing mobile development are starting to seriously look at some particular Clojure implementation. Who knows what interesting tidbits might rise to the surface if someone really dug into the data…

Finally, please do what you can to spread around this survey to those that you know of that are working with Clojure — really, in any capacity.  You’ll find various social media chicklets at the bottom of this post if you want the lazy way out.

The survey is now closed.  I’ll be posting the results and my pithy analysis sometime next week…

Posted in Clojure | 4 Comments

On the stewardship of mature software

I just flipped the switch on v2.5.0 of PDFTextStream.  It’s a fairly significant release, representing hundreds of distinct improvements and bugfixes, most in response to feedback and experiences reported by Snowtide customers.  If you find yourself needing to get data out of some PDF documents, you might want to give it a look…especially if existing open source libraries are falling down on certain documents or aren’t cutting it performance-wise.

But, this piece isn’t about PDFTextStream, not really.  After prepping the release last night, I realized that PDFTextStream is ten years old, by at least one reckoning: though the first public release was in early 2004, I started the project two years prior, in early 2002, ten years ago. Ten years.

It’s interesting to contemplate that I’m chiefly responsible for something that is ten years old, that is relied upon by lots of organizations internally, and by lots of companies as part of their own products.  Aside from the odd personal retrospectives that can be had by someone in my situation (e.g. friends of mine have children that are around the same age as PDFTextStream; am I better or worse off having “had” the latter when I did instead of a son or daughter?), some thought has to be given to what the longevity and particular role of PDFTextStream (or, really, any other piece of long-lived software) implies and requires.

I don’t know if there are any formal models for determining the maturity of a piece of software, but it seems that PDFTextStream should qualify by at least some measures, in addition to its vintage.  So, for your consideration, some observations and opinions from someone that has their hand in a piece of mature software:

Mature software transcends platforms and runtimes

PDFTextStream is in production on three different classes of runtimes: all flavours of the JVM, both Microsoft and Mono varieties of the .NET CLR, and the CPython implementation of Python.  This all flows from a single codebase, which reminds me many kinds mature systems (sometimes referred to as “legacy” once they’re purely in maintenance mode — a stage of life that PDFTextStream certainly hasn’t entered yet) that, once constructed, are often lifted out of their original runtime/platform/architecture to sit on top of whatever happens to be the flavour of the month, without touching the source tree.

Often, the effort required to make this happen simply isn’t worth it; the less mature a piece of software is, the easier it is at any point to port it by brute force, e.g. rewriting something in C# or Haskell that was originally written in Java.  This is how lots of libraries made the crossing from the JVM to .NET (NAnt and NHibernate are two examples off the top of my head).

However, the more mature a codebase, and the more challenging the domain, the more unthinkable such a plan becomes. For example, the prospect of rewriting PDFTextStream in C# to target .NET — or, if I had my druthers, rewriting PDFTextStream in Clojure to satisfy my geek id — is absolutely terrifying.  All those years of fixes and tweaks in the PDFTextStream sources…trying to port all of them to a new implementation would constitute both technical and business suicide.

In PDFTextStream’s case, going from its Java sources to a .NET assembly is fairly straightforward given the excellent IKVM cross-compiler.  However, there’s no easy Java->Python transpiler to reach for, and a bytecode cross-compiler wasn’t available either.  The best solution was to invest in making it possible to efficiently load and use a JVM from within CPython (via JNI).  With that, PDFTextStream, derived from Java sources, ran without a hitch in production CPython environments. Maybe it was a hack, but it was, in relative terms, easier and safer than any alternative, and had no downsides in terms of performance or capabilities.

(I eventually nixed the CPython option a few years ago due to a lack of broad commercial interest.)

Thou shalt not break mature APIs

When I first started programming in Java, I sat aghast in the ominous glow of java.util.Date. It was a horror then, and remains so. The whole thing has been marked as deprecated since 1997; and, despite the availability of all sorts of better options, it has not been removed from the standard library.  Similar examples abound throughout the JRE, and all sorts of decidedly mature libraries.

For some time, I attributed this to sloth, or pointy-haired corporate policies, or accommodation of such characteristics amongst the broad userbase, or…god, I dunno, what are those guys thinking? In the abstract, if the physician’s creed is to “do no harm”, it seems that the engineer’s should be “fix what’s broken”; so, continual improvement should be the law of the land, API compatibility be damned.

Of course, it was naïve for me to think so.  Brokenness is often in the eye of the beholder, and formal correctness is a rare thing outside of mathematics.  Thus, the urge one has to “make things better” must be tempered by an understanding of the knock-on effects for whoever is living downstream of you.  In particular, while making “fixes” to APIs that manifest breaking changes — either in terms of signatures or semantics — might make you feel better, there are repercussions:

  • You’ll absolutely piss off all of your customers and users.  They had working code that now doesn’t work. Whether you are charging them money or benefiting from their trust, you are now asking them to take time out of their day to help you feel better about yourself.
  • Since their code is broken already, your customers and users might see this as the perfect opportunity to make their own changes to not have to cope with your self-interested “fixes” anymore.  Surely you can imagine the scene:

    Sarah: “Hey Gene, the new version of FooLib changes the semantics of the Bar(string) function. Do you want me to fix it now?”

    Gene: “Sheesh, again? Well, weren’t you looking at BazLib before?”

    Sarah: “Yeah; BazLib isn’t quite as slick, but Pete over in Accounts said he’s not had any troubles with it.”

    Gene: “I’m sold. Stick with the current version of FooLib for now, but next time you’re in that area of the code, swap it out for BazLib instead.”

This is why semantic versioning is so important: when used and understood properly, it allows you to communicate a great deal of information in a single token.  It’s also why I can often be found urging people to make good breaking changes in v0.0.X releases of libraries, and why PDFTextStream hasn’t had a breaking change in 6 years.

Of course there are parts of PDFTextStream’s API that I’m not super proud of; I’ve learned a ton over the course of its ten year existence, and there are a lot of things I’d do differently if I knew then what I know now.  However, overall, it works, and it works very well, and it would be selfish (not to mention a bad business decision) to start whacking away at changes that make the API aesthetically more pleasant, or of marginally higher quality, but which make customers miss a beat.

It seems to me that a good guideline might be that any breaking change needs to be accompanied by a corresponding 10x improvement in capability in order to be justifiable.  This ties up well with the notion that a product new to the market must be 10x better than its competition in order to win; insofar as a new version of the same product with API breakage can potentially be considered as foreign as competing products, that new version is a new product.

Managing risk is Job #1

If your hand is on the tiller of some mature software — or, some software that you would like to see live long enough to qualify as mature — your first priority at all times is to manage, a.k.a. minimize, risk for your users and customers.

As Prof. Christensen might say, software is hired to do a job.  Now, “managing risk” isn’t generally the job your software is hired to do, e.g. PDFTextStream’s job is to efficiently extract content from any PDF document that is thrown at it, and do so faster and more accurately than the other alternatives.  But, implicit in being hired for a job is not only that the task at hand will be completed appropriately, but that the thing being hired to do that job doesn’t itself introduce risk.

The scope of software as risk management is huge, and goes way beyond technical considerations:

  • API risk, as discussed above in the “breakage” section
  • Platform risk. Aside from doubling the potential market for PDFTextStream, offering it on .NET in addition to the JVM serves a purpose in mitigating platform risk for our customers on the JVM: they know that, if they end up having to migrate to .NET, they won’t have to go find, license, and learn a new PDF content extraction library.  In fact, because PDFTextStream licenses are sold in a platform-agnostic way, such a migration won’t cost a customer of ours a penny.  Of course, the same risk mitigation applies to our .NET customers, too.
  • Purchasing risk. Buying commercial software outside of the consumer realm can be a minefield: tricky licensing, shady sales tactics, pricing jumping all over the map (generally up), and so on.  PDFTextStream has had one price increase in eight years, and its licensing and support model hasn’t changed in six.  Our pricing is always public, as is our discount schedule.  When one of our customers needs to expand their installation, they know what they’re getting, how much it’s going to cost, and how much it’ll cost next time, too.

Even if one is selling a component library (which PDFTextStream essentially is), managing risk effectively for customers and users can be a key way to offer a sort of a whole product.  Indeed, for many customers, managing risk is something that you must do, or you will simply never be hired for that job, no matter how well you fulfill the explicit requirements.

Posted in Business, Craftsmanship, PDFTextStream | Leave a comment

What sucks about Clojure…and why you’ll love it anyway

I gave a talk with this title at the wonderful Clojure/West conference back in March.  The video and slides for it are now available on InfoQ here.

You can go watch the talk itself, so I won’t repeat its preface.  I’ll just say that, having seen the talk for the first time myself, I’m very happy with its content and delivery.  I’m hoping that others — especially those new to Clojure or looking at Clojure for the first time — will find the talk helpful in directing their attention to the tricksier bits of the language.

Posted in Clojure | 1 Comment

Starting Clojure (mk. 2)

I’ve wanted to put together a long-form introductory Clojure screencast for some time.  I had an opportunity to do this in grand style yesterday in a live O’Reilly webcast, but, for various reasons, I wasn’t fond of how that came together.  So, I cut another live coding screencast that introduces, in various levels of detail:

A complete (though not pretty!) URL shortener webapp is built from scratch, with discussions of immutable data structures, function composition, and mild concurrency topics scattered throughout.  It’s a fair bit more than I was planning on covering in the O’Reilly webcast, but I think the additional material blended in well.

Without further ado, Starting Clojure, mk. 2:

(You may want to watch full screen in HD to see various details)

Once you’re done with the screencast, you may want to continue your Clojure explorations with the help of Clojure Programming, and maybe Clojure Atlas (which, conveniently enough, is available at a hefty discount with your copy of  Clojure Programming).

Backstory on the O’Reilly Starting Clojure webcast

Ever had a bad day? Sure, of course. Ever had a really bad day prior to presenting a live-coding webcast to what turned out to be ~700 internet attendees?  Yeah, that was me yesterday. If you’re brave (or want to wince, laugh, and then cry at my performance), you will likely be able to see the video of it eventually. But seriously, don’t bother.

I probably should have postponed the whole thing, but that seemed unreasonable at the time — the fact that it had been planned for a couple of weeks, had a bunch of registered attendees, and my own stubbornness urged me on to commit programmer seppuku.  I was existentially distracted the whole time, and the more I tried to hold it together, the worse things got.  (Of course, that’s not an excuse, but an explanation.)  Honestly, after finishing the webcast, I was absolutely horrified; I had a great opportunity to represent Clojure well to a large body of programmers new to the language, and I utterly failed.  I felt like I had done a disservice to O’Reilly and, most of all, my coauthors.

Fight or flight kicked in, and for 5 minutes, I harbored thoughts of giving up doing screencasts and public speaking permanently, to save everyone involved. Thankfully, I relaxed, had a couple glasses of wine, and woke up early this next morning with a clear head to record a live-coding screencast, in proper single-take style, which you see above.  It is epically better than the O’Reilly webcast, covers the material better than I planned, and was marred only by a couple of minor hiccups that were more funny than sad.

That is to say, mk. 2 is entirely in keeping with my usual baseline, and I’m happy to have it out there.  In the end, I hope more people see it than the first webcast I did.  In any case, I’m glad to have gotten back on the horse and hopefully redeemed myself by some measure.

Posted in Clojure, Clojure Programming (book) | 28 Comments

A refresh of Clojure Atlas

I’m sorry to admit that I let the Clojure Atlas wilt a bit over the past year or so. (I was a little busy!)  However, I am conversely quite happy to say that that’s over now; Clojure Atlas has been refreshed to add editions for Clojure v1.3.0 and v1.4.0.

(If you don’t know what Clojure Atlas is, head on over and check out the snazzy new demo/tour video.)

Other highlights include:

Pricing changes

I think the previous pricing was too high.  (You never know until you try.)  Pricing has been lowered, and I’ve added a fun option whereby you can get any edition of Clojure Atlas for just $5.  I don’t quite know what I’ll end up doing for upgrades going forward, but you will definitely be able to stay current without paying the full boat each time.

Free upgrades

Between the too-high pricing and the far-too-long period between the initial release of Clojure Atlas and now, those that prepaid for access to the Clojure v1.3.0 Atlas how have access to all of them, up to and including v1.4.0.  Those early significant supporters will also get free upgrades to all future Clojure Atlas revisions.  Thanks, guys and gals.

If you only purchased the Atlas for Clojure v1.2.0 previously, your account has been upgraded to include the Atlas for v1.3.0.

Ontology improvements

Aside from the obvious additions that needed to go in to reflect changes in Clojure v1.3.0 and v1.4.0, the ontology has been improved significantly to be more comprehensive and more accurate.  In addition, I’ve started adding detailed documentation (for example) to subjects/nodes within the ontology that I’ve added (in contrast to vars, which in general already have documentation of their own).

Visualization improvements

The graph visualization is certainly far from perfect, but I’ve tweaked it a fair bit to get it to “settle” faster than it did before.  I’m also pondering a complete reworking of the visualization to make it deterministic (rather than using a particle simulation as it does now).

No more PayPal

Many people balked at using PayPal — and believe me, no one is happier than I to be rid of it at this point.  Payments are now all handled courtesy of Stripe, which has been a dream to work with.

Posted in Announcements, Clojure, Clojure Atlas | Leave a comment

Friend: an extensible authentication and authorization library for Clojure Ring webapps and services

Say hello to my little Friend.

There’s plenty of technical stuff in the README to chew on if you like.  In short, I’m hoping this can eventually be a warden/spring-security/everyauth /omniauth for Clojure; that is, a common abstraction for authentication and authorization mechanisms.  Clojure has been around long enough that adding pedestrian things like form and HTTP Basic and $AUTH_METHOD_HERE to a Ring application should be easy.  Right now, it’s not: either you’re pasting together a bunch of different libraries that don’t necessarily compose well together, or you get drawn into shaving the authentication and authorization yaks for the fifth time in your life so you can sleep well at night.

Hopefully Friend will make this a solved problem, or at least push things in that direction.  It plays nice with all of the best principles of Ring, and includes support for:

  • form, HTTP Basic, and OpenID authentication
  • role-based authorization (optionally using hierarchical roles via Clojure’s derive and isa?)
  • su capabilities (multiple login support / a.k.a. “log in as”)
  • channel security (i.e. HTTPS-only for certain Ring routes)
  • …and more

Most importantly, it takes a stab at a couple of core abstractions for others to drop in other authentication workflows, e.g. OAuth in all of its incarnations, NTLM, BrowserID, etc. etc. etc.  There are already plenty of Clojure implementations for all sorts of authentication methods; hopefully someone (you?!) will step up and bring one of them to the party, so anyone’s Friend-empowered Clojure webapp can easily offer any or all of them with a minimum of suffering.

Finally: frankly, it’s absurd that I’m writing security-related stuffs.  (I know it hardly ever works out that way, but it seems like some experts somewhere should be taking care of this.)  It would be a great thing if you were to beat on Friend and try to find exploits, general breakage, etc., especially if you have prior experience in this area.

Posted in Announcements, Clojure, Open Source | 4 Comments

Originally posted on Mostly λazy…a Clojure podcast:

Recorded November 12th, 2011, the fourth and final recording in a series of conversations from Clojure Conj 2011.

Chris Houser (usually known as chouser online) has been working with Clojure longer than nearly anyone else; he started tinkering with the language in early 2008, and was a fixture in #clojure irc and on the mailing list for years.  His contributions to the language, early libraries, and community through his always genial and insightful presence are hard to overstate.  More recently, he has coauthored the excellent Joy of Clojure along with Michael Fogus, and is now working with Clojure daily over at Lonocloud.

It’s been my privilege to know and work with Chris a bit over the years, and, as always, it was great to talk with him in person.

Enjoy!

Listen:


Or, download the mp3 directly.

Discrete Topics

  • “Everything I learned, I [learned] on irc?!”
  • Macros…

View original 111 more words

Posted in Uncategorized | Leave a comment

‘Clojure Programming’ book finished

Yes — it’s finished! :-D

Early last month, after writing 190,000 words, editing away scads more, assembling and testing more than 1,000 code snippets and 20 full sample projects, and conceptualizing dozens of illustrations, Christophe, Brian, and I declared Clojure Programming done.  It’s been writhing its way through O’Reilly’s editorial process ever since.

I’d hoped that the book would be published before Clojure/West in mid-March, but alas, it was not to be.  It looks like it’ll drop in mid-April.

However, fret not! If you want to dig into Clojure Programming right away, you can read the final draft of it online.  Of course, you can preorder the dead-tree version of it as well; easy links to both options are available at clojurebook.com.  There, you’ll also find a full table of contents, some basic info on the book, and a way to join the clojurebook.com mailing list and a pointer to the book’s Twitter account.  We’ll be pushing various Clojure tips and links to useful tools and resources and announcing the availability of all sorts of book-related content on the site through the mailing list and Twitter feed; and, if things work out as I hope, some early access to and/or special offers for things that will help you get the most out of your Clojure experience in general.

So, thanks for your patience.  I think the book will end up being worth it.  Of course, I have to thank my coauthors; without Brian and Christophe, it simply would never have been finished, nor would it be as good as it is.  There’s a ton of other people that deserve credit too, but you’ll have to buy the book and read the acknowledgements to learn about them…

Posted in Books, Clojure, Clojure Programming (book) | 5 Comments

Originally posted on Mostly λazy…a Clojure podcast:

Recorded November 12th, 2011, third in a series of conversations from Clojure Conj 2011.

I caught up with Hugo Duncan and Antoni Batchelli (everyone calls him Toni ;-) during one of the lunch breaks at the Conj.  These guys have been on a tear with Pallet, an open source Clojure project that Hugo started in early 2010 to shave one of the hairiest yaks around, the automation of provisioning and management of computing infrastructure.  The result is a tool and library that provides a classically Clojure abstraction for controlling nearly any environment, from cloud nodes to virtual machines to the rackmounts you have downstairs.  Since it is their full-time job — Toni and Hugo have built a business around the project — most of our discussion centers on Pallet, its history, and how people are using it.

Near the end, Hugo and I talk some about his other…

View original 270 more words

Posted in Uncategorized | Leave a comment

Introducing: Mostly λazy (a Clojure podcast)

As some of you may know already, I’ve started a Clojure podcast called Mostly λazyThe first episode is in the can and has been published.  There are RSS feeds for your readers and podcatchers there for the taking, and I’ll be working on getting the feed into iTunes shortly.

The initial announcement post for Mostly λazy drew a lot of really good ideas and suggestions, many of which I’ll be stealing. :-D I’m really glad that a lot of people find the concept of a Clojure podcast appealing; whilst I’m fundamentally doing this to enjoy myself, it’s nice to know that there will be at least a dozen or so people listening… ;-)

I’ll be at Clojure Conj 2011 this week, and will likely capture a bunch of conversations with people there.  That should provide me with a decent backlog of material that I can trickle out through the rest of the year, perhaps mixed in with some shorter segments based on the aforementioned suggestions.  In any case, I hope anyone who does listen finds the experience entertaining and engaging.

By the way, this is likely the first and last post I’ll make here about Mostly λazy. It has its own site/blog, so you should go subscribe there if you want to see more Clojure podcast news in the future.

Posted in Clojure | Leave a comment