Flowchart for choosing the right Clojure type definition form

Clojure offers a number of different forms that define types (and generate Java classes). Choosing between deftype, defrecord, reify, proxy, and gen-class can be a tripping point for those new to Clojure.

I’d obviously like to make such decisions easier for everyone.  I know that many people learn best through visual aids, so I’ve been working on a flowchart that attempts to encapsulate some of the significant choices that go into deciding between the different type-definition forms.  A draft of it is below; let me know if it is helpful to you (or not!), how you think it could be made better, etc.  The most recent version of the flowchart, the “source” OmniGraffle file, and many translations of it (feel free to contribute another one!) will always be available here.


“The ‘Interop Zone'” demarcates use cases (e.g. needing to define multiple constructors) and forms (proxy and gen-class) that are exclusively the domain of Clojure’s JVM interoperability support. Using these forms is fine, but be aware that by doing so, you are stepping outside of Clojure’s “native” abstractions; unless you are defining a type for the express purpose of meeting interoperability requirements, one of Clojure’s simpler type-definition forms may suit your needs better.


Updated 2011-07-05 18:28 UTC based on initial feedback. Thanks!

Updated 2011-07-07 17:38 UTC — added “The ‘Interop Zone'”

Updated 2011-08-09 16:30 UTC — Japanese translation provided by OGINO Masanori

Updated 2011-08-23 17:33 UTC — Portuguese translation provided by Paulo Suzart

Final update: The most recent version of the flowchart, the “source” OmniGraffle file, and many translations of it (feel free to contribute another one!) will always be available here.

The 2011 State of Clojure Survey is Open

I’ve finished putting together the State of Clojure 2011 survey, which is embedded below.  It will remain open for input until Monday, June 20th.

As was the case last year, we are likely on the verge of a new “major” release of Clojure (v1.3.0 this time around), and the apparent state of affairs can be described similarly:

The Clojure community is larger than it ever has been, and shows no sign of slackening its growth.  It seems like now would be a good time to take stock of where the community is, how people came to use Clojure, and how it’s being used in the world.

Hopefully enough responses will come through that we’ll be able to get a good picture of the current state of affairs, and maybe a little insight into where Clojure can and should make headway in the future.

As with the State of Clojure 2010 survey, I will post again sometime shortly after the survey closes with all of the captured data, some pretty charts, and whatever attempts at witty, (un?)biased commentary I can come up with. ;-)  Seriously though, we’ll have prior data available this time around, so some interesting trends and comparisons might be inferred.

The survey has now ended.

A summary of results and all the raw data are now available here.

‘Clojure Programming’ book now available

Update [2011-08-23 18:49 UTC]: The Rough Cut of Clojure Programming has been updated significantly since this post originally went live.  Go check it out. :-)

Some time ago, I announced that I was coauthoring a book on Clojure for O’Reilly (see original announcement).  I’m very happy to report that an early and incomplete version of Clojure Programming is now available in Rough Cuts.

Rough Cuts is O’Reilly’s early-access program, similar to Manning’s MEAP.  By purchasing it now, you will be able to read the ebook via Safari as it progresses through its final stages, and leave feedback that we will take into account through that process.  Please make use of the comment/feedback facility on the book’s Safari page; we are eager to hear what you have to say about the book — though personally, I vacillate between hoping you’ll be gentle and hoping you’ll be brutal.

What’s in the first Rough Cut is actually the state of the book from about two months ago.  I dropped the ball on giving the final word to our editor to go ahead with the release, so I’m afraid you’re all getting this much later than you could (and should) have.  On the upside, there’s a lot of content queued up to be added to the Rough Cut, so you’ll be seeing new stuff stream in very rapidly from here on out.

I do want to apologize about (inadvertently) maintaining radio silence about the book since my original announcement.  Writing the book has ended up overlapping with a very busy time in my life, and I needed to recruit new coauthors mid-stream to boot.  Dave had some killer opportunities that he simply couldn’t turn down; his departure was unfortunate, but it gave me the great opportunity to work with two very well-known figures in the Clojure community:

  • Brian Carper, a stellar writer (I’d been a fan of his blog for some time) and former Ruby hacker (a perspective I wanted to make sure we serviced in the book well)
  • Christophe Grand, the author of a host of popular Clojure libraries such as Enlive, Parsley, and Moustache, and blogger of all things bleeding-edge in Clojure

I’m biased of course, but the book is shaping up to be what I think will be a great introduction to Clojure — especially for those coming from Java, Ruby, and Python — and simply none of it would have been possible if it were not for Brian and Christophe.  Thanks, guys! :-D

Preview and purchase the book: Clojure Programming

P.S. I just want to take a moment to let it settle in that, yes, O’Reilly is publishing a Lisp book, despite their explicitly discouraging Lisp topics in their book proposal guidelines.  (Sorry guys, a single friendly needling is warranted. ;-)) I know it’s not an old concept (they accepted our proposal, after all, and then there was the sadly ill-fated Lisp: Out of the Box), but now the bits are flowing, orders are being taken, and it can’t get much more official. Happy days indeed.

State of Clojure, 2011: What questions do you have?

Update: The 2011 State of Clojure survey is now open.

I just realized that I’m running a tad behind in setting up this year’s State of Clojure survey!

The first State of Clojure survey, which I ran in June of last year, provided a useful view of how Clojure was being used, what directions the Clojure community wanted to see the language move in, and who the community was in very broad terms.

I’m now setting up for this year’s survey, and wanted to ask: what questions do you think I should ask?

For reference, I asked ten questions last year (see the results post for what the various answers were):

  1. For how long have you been using Clojure?
  2. How would you characterize your use of Clojure *today*?
  3. What is the status of Clojure in your workplace?
  4. What language did you use just prior to adopting Clojure — or, if Clojure is not your primary language now, what is that primary language?
  5. What do you think is Clojure’s most glaring weakness / blind spot / problem?
  6. If Clojure disappeared tomorrow, what language(s) might you use as a “replacement”?
  7. In which domain(s) are you using Clojure?
  8. Clojure is primarily a JVM-hosted language.  Which other platform(s) would you be interested in using Clojure on, given a mature implementation?
  9. Which environment(s) (i.e. IDEs/editors) do you use to work with Clojure?
  10. What questions would you like to see added?  What answers would you like to see added to the old questions?

I’m planning on starting the survey next Monday, the 13th of June, so get your suggestions in early!

Clojure Atlas now available

A few weeks ago, I previewed Clojure Atlas; I’m happy to announce that it is now publicly available:

Clojure Atlas, an experimental visualization of the Clojure language and its standard library

There’s a free demo available which will nag you after a bit; if you find it useful, interesting, helpful, or even just a little fun, it would be great if you purchased Clojure Atlas for whichever version(s) of Clojure you’re interested in.

For some limited period of time, you can pre-order an Atlas for the forthcoming Clojure v1.3 at a $10 discount, so grab that soon if you are interested in it.

Please note that Clojure Atlas is by no means Done, or even “done”.  I’m opening it up for use and purchase now in the spirit of release early, release often, so you are sure to find rough edges in the UI and straightforward incompleteness in the ontology that drives the Atlas.  You can read more about the current status here.

As for what’s in the future for Clojure Atlas, my personal to-do list is far too long to go into.  Part of why I’m releasing it “early” is to get a sense of what people want, and what will be most useful.  Doing otherwise would surely lead me to fritter away precious days and weeks honing features interesting to only a few.  There are links all over the site and in the Atlas itself where you can submit ideas, suggestions, and bug reports via all sorts of channels; I’ll be keeping a close eye on the email, tweets, and UserVoice threads submitted in order to filter and prioritize future work.

Why are you still reading this post?  Go check out the Clojure Atlas!

P.S. I’d like to thank all of the early-access testers that provided valuable feedback leading up to this release.  In particular, Edmund Jackson spent far more time with me on irc than he needed to, helping to ferret out issues in the earlier revisions of the graph visualization, and making a variety of excellent suggestions for future development.

Adding Gzip compression to a Clojure webapp in 30 seconds

As you might have seen, I’m working on a new web project, which happens to involve shipping a metric ton of content to each user’s browser upon visiting the meat of the site.  We’re talking about something like 1.5MB of HTML, Javascript, and CSS, and that’s after best-effort minification and such.

Clearly, Gzipping the whole mess is called for.  I’ve never worked on any high-volume sites that called for such measures, so this is a new requirement for me.  The site’s backend is implemented in Clojure though, so my first instinct was to Google “gzip ring clojure” (Ring being the thoroughly spectacular Clojure web framework), whereupon I found Michael Stephens’ ring-gzip-middleware project.  Seems simple enough: Ring request handlers are just functions, and you can apply middleware trivially via function composition, so ring-gzip-middleware provides a function that wraps your assembled Ring handler(s) to compress outgoing responses appropriately.

Thankfully, I stopped short just in the nick of time: while Ring middleware is a damn fine hammer, Gzip response compression shouldn’t be a nail in this scenario.  I didn’t want to have to read through entirely unrelated infrastructure-related bits in my codebase henceforth, no matter how elegantly they folded themselves in.  There remains value in the notion of separation of concerns.

Again thankfully, Clojure web apps are Java web apps, universally deployed in a servlet container like Tomcat, Jetty, Glassfish, and so on. So, I have quite the tasty menu to choose from.

Container-provided Gzip compression

Most if not all Java servlet containers provide Gzip response compression out-of-the-box.  Tomcat, for example, requires simply adding a couple of attributes to the Connector element in its server.xml file.  Jetty handily provides Gzip compression of static resources via its default servlet; just set its gzip init parameter to true in your web.xml1 file, and you’re done:

<servlet>
    <servlet-name>default</servlet-name>
    <servlet-class>org.eclipse.jetty.servlet.DefaultServlet</servlet-class>
    <init-param>
        <param-name>gzip</param-name>
        <param-value>true</param-value>
    </init-param>
</servlet>

I think you can leave out the <servlet-class> element there; I didn’t experiment much on this path, because:

  1. I needed to be able to Gzip dynamically-generated content, and
  2. I use Jetty in my development environment, but deploy to Tomcat in production, so I wanted a general-purpose solution.

Thus, what I consider to be the ideal approach:

Gzip Servlet Filters

Servlet filters are independent, composable components that can dynamically modify requests and responses, the circa-1999 Java corollary to Ring middleware.  The difference is largely in packaging and context: while Ring middleware is just a function that can be folded into a codebase programmatically, servlet filters are specified statically as part of an application’s web.xml file, and in general are not modified from within the servlet at runtime.  (Things can get interesting when you implement servlet filters using Clojure, but that’s perhaps a topic for another post.)

There are various Gzip compression servlet filter implementations floating around the ‘nets, including one particularly bad example that appeared in some magazine in 2004 that has an unreasonable amount of Google juice associated with it for some reason.  Use any of them, and your Clojure web application will be Gzip-ready, regardless of which servlet container you deploy to.  For my money, the best one is provided by the Jetty project, simply because it’s impossible to argue with its provenance given that Jetty is used everywhere: if there was a problem with its Gzip servlet filter implementation, it certainly would’ve been found out by now.

Using it is cake; add the corresponding dependency to your pom.xml file:

<dependency>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty-util</artifactId>
    <version>6.1.26</version>
</dependency>

and add it to your web.xml file with a corresponding URL mapping:

<filter>
    <filter-name>jetty-gzip</filter-name>
    <filter-class>org.mortbay.servlet.GzipFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>jetty-gzip</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

Done.

All my content is now Gzip-compressed, both dynamically-generated and static (because I always use the servlet container’s default servlet for serving up static resources).  I didn’t have to make any changes to my codebase, and I’ll never once be reminded of Gzip compression when I fiddle with my Ring handlers.

The Jetty Gzip filter has various options for tweaking which mime types and sizes of content should be included and for excluding specific user agents from receiving Gzipped content, but I’ll just leave the defaults alone for now (i.e. compress everything for everyone).

Postscript: Wait, what’s with all the XML in this “Clojure web app”?

Some people are allergic to parentheses; some are allergic to XML; I choose to find peace in both as appropriate. :-)

That Jetty Gzip filter does a good job of something I don’t want to think about.  Just as core vs. context is a useful frame in business affairs, I think it’s handy when thinking about how to approach software development.  I don’t get points for having a “pure Clojure” stack for my web application, especially if I need to break away from having a reasonable separation of concerns or deal with FUD creeping up in my head about whether a reimplementation of fundamentally commodity operations is really up to spec or not — it may very well be, but there’s simply no gain to be had when that choice pans out favorably if there are safer alternatives for such matters of context.

Thus, I chose the ~5-year-old Gzip filter from Jetty, just like I often choose to use boringly reliable Java-land libraries (e.g. Spring Security) and tools (e.g. Maven and Eclipse) to support far more interesting things for which I use bleeding-edge kit like Clojure.

Footnotes
  1. You know what a web.xml file is, right?  Every Java web application has one, whether it’s generated by your build process or you create it yourself.  The latter is generally preferable IMO, simply because you can take advantage of all the goodies that it opens up for you.  You can read generalities about web.xml files all over the web; there’s a sample Clojure web project over here that contains a simple example, and I talk about them a bit in my post and screencast here.

Clojure Atlas (Preview!)

Today, I’m opening up a “preview” site for Clojure Atlas, a new side project of mine that I’m particularly excited about.

Clojure Atlas is an experiment in visualizing a programming language and its standard library.  I’ve long been frustrated with the limitations of text in programming, and this is my attempt to do something about it.  From the site:

While Clojure Atlas has a number of raisons d’être, it fundamentally exists because I’ve consistently thought that typical programming language and API references – being, in general, walls of text and alphabetized links – are really poor at conveying the most important information: not the minutiae of function signatures and class hierarchies, but the stuff that’s “between the lines”, the context and interrelationships between such things that too often are only discovered and internalized by bumping into them in the course of programming. This is especially true if we’re learning a language and its libraries (really, a never-ending process given the march of progress), and what’s standing in our way is not, for example, being able to easily access the documentation or signature for a particular known function, but discovering the mere existence of a previously-unknown function that is perfect for our needs at a given moment.

This is just a preview – all sizzle and no steak, as it were.  I’m working away at the ontology that drives the visualization and user experience, but I want to get some more early (quiet) feedback from a few folks to make sure I’m not committing egregious sins in various ways before throwing open the doors to the world.

In the meantime, if you’re really interested, follow @ClojureAtlas, and/or sign up for email updates on the site.

Minifying CSS with Clojure

Updated: Thanks to Alan Malloy for poking me about the comment-related regular expression in the snippet below being too greedy.

A recent project had me looking into minifying CSS.  This is unfamiliar territory for me since I’ve rarely cared much about the size of website assets.  In the beginning of the project, minifying the Javascript that was floating around was more of a priority, so I folded Google Closure into my build process (which beat out YUI’s Javascript minification by a substantial margin for my codebase – though I was quite disappointed that Google Closure’s advanced compilation appears to be unusable by mere mortals).  So, when I needed to also minify CSS (something Closure doesn’t do), I didn’t want to fritter away more time messing with YUI as well.

Thus, I reinvented the wheel – something I am usually extraordinarily reluctant to do, but this was simply too easy to pass up:

(require 'clojure.string)

(defn minify-css
  "Minifies the given CSS string, returning the result.
   If you're minifying static files, please use YUI."
  [css]
  (-> css
    (clojure.string/replace #"[\n|\r]" "")
    (clojure.string/replace #"/\*.*?\*/" "")
    (clojure.string/replace #"\s+" " ")
    (clojure.string/replace #"\s*:\s*" ":")
    (clojure.string/replace #"\s*,\s*" ",")
    (clojure.string/replace #"\s*\{\s*" "{")
    (clojure.string/replace #"\s*}\s*" "}")
    (clojure.string/replace #"\s*;\s*" ";")
    (clojure.string/replace #";}" "}")))

I’m guessing this wouldn’t fare well with the various CSS syntax tricks sometimes played by those having to accommodate multiple archaic versions of IE, but itworksforme. Caveat emptor, in any case.

It can be very handy to have minification routines available at runtime without digging into the frightening APIs of either Closure or YUI, both of which are intended to be used only from the command line.  Sometimes assets are generated dynamically, and sometimes it’s handy to be able to drop them straight into outgoing HTML pages (especially so if you want to produce a standalone version of a page, for example).

There’s scads of CSS minification snippets around that use regular expressions, so this is nothing special, but I thought some Clojure web developers might appreciate it.

Bandalore: a Clojure client library for Amazon’s Simple Queue Service (SQS)

I recently found myself wanting to work with Amazon’s Simple Queue Service (SQS), but I could find no reasonable Clojure library for accessing it.  Of course, AWS’ own Java SDK is the canonical implementation of their APIs (at least in the JVM space), so putting together a Clojure wrapper that adds a few handy extras wasn’t particularly difficult.

You can find Bandalore hosted on github, licensed under the EPL. A proper release will find its way into Maven central within the next couple of days.  The code isn’t much more than 12 hours old, so consider yourself forewarned. ;-)

I hope people find the library useful.  If you’ve any questions, feel free to ping me in irc or twitter.

What follows is an excerpt from the README documentation for Bandalore that describes some of its more interesting functionality:


seqs being the lingua franca of Clojure collections, it would be helpful if we could treat an SQS queue as a seq of messages. While receive does return a seq of messages, each receive call is limited to receiving a maximum of 10 messages, and there is no streaming or push counterpart in the SQS API.

The solution to this is polling-receive, which returns a lazy seq that reaches out to SQS as necessary:

=> (map (sqs/deleting-consumer client :body)
     (sqs/polling-receive client q :limit 10))
("3" "5" "7" "8" ... "81" "90" "91")

polling-receive accepts all of the same optional kwargs as receive does, but adds two more to control its usage of receive API calls:

  • :period – time in ms to wait after an unsuccessful `receive` request (default: 500)
  • :max-wait – maximum time in ms to wait to successfully receive messages before terminating the lazy seq (default 5000ms)

Often queues are used to direct compute resources, so you’d like to be able to saturate those boxen with as much work as your queue can offer up. The obvious solution is to pmap across a seq of incoming messages, which you can do trivially with the seq provided by polling-receive. Just make sure you tweak the :max-wait time so that, assuming you want to continuously process incoming messages, the seq of messages doesn’t terminate because none have been available for a while.

Here’s an example where one thread sends a message once a second for a minute, and another consumes those messages using a lazy seq provided by polling-receive:

=> (defn send-dummy-messages
     [client q count]
     (future (doseq [n (range count)]
               (Thread/sleep 100)
               (sqs/send client q (str n)))))
#'cemerick.bandalore-test/send-dummy-messages
=> (defn consume-dummy-messages
     [client q]
     (future (dorun (map (sqs/deleting-consumer client (comp println :body))
                      (sqs/polling-receive client q :max-wait Integer/MAX_VALUE :limit 10)))))
#'cemerick.bandalore-test/consume-dummy-messages
=> (consume-dummy-messages client q)               ;; start the consumer
#<core$future_call$reify__5500@a6f00bc: :pending>
=> (send-dummy-messages client q 1000)             ;; start the sender
#<core$future_call$reify__5500@18986032: :pending>
3
4
1
0
2
8
5
7
...

You’d presumably want to set up some ways to control your consumer. Hopefully it’s clear that it would be trivial to parallelize the processing function being wrapped by deleting-consumer using pmap, distribute processing among agents if that’s more appropriate, etc.

How should Clojure applications be structured?

Random thought from #clojure irc last night:

Clojure disposes of so much of the ceremony that exists in most programming languages and is so “data-oriented” that the structure of Clojure applications seems to represent the contours one’s domain far more than any other language I’ve encountered.

That is to say, outside of some probably very broad idioms, there’s probably no such thing as a “typical structure” of Clojure programs.