Reblogged from Mostly λazy…a Clojure podcast:

Recorded November 12th, 2011, third in a series of conversations from Clojure Conj 2011. I caught up with Hugo Duncan and Antoni Batchelli (everyone calls him Toni ;-) during one of the lunch breaks at the Conj.  These guys have been on a tear with Pallet, an open source Clojure project that Hugo started in early 2010 to shave one of the hairiest yaks around, the automation of provisioning and management of computing infrastructure.  The result is a tool and library that provides a classically Clojure …

The initial announcement post for Mostly λazy drew a lot of really good ideas and suggestions, many of which I’ll be stealing. :-D I’m really glad that a lot of people find the concept of a Clojure podcast appealing; whilst I’m fundamentally doing this to enjoy myself, it’s nice to know that there will be at least a dozen or so people listening… ;-)

I’ll be at Clojure Conj 2011 this week, and will likely capture a bunch of conversations with people there.  That should provide me with a decent backlog of material that I can trickle out through the rest of the year, perhaps mixed in with some shorter segments based on the aforementioned suggestions.  In any case, I hope anyone who does listen finds the experience entertaining and engaging.

By the way, this is likely the first and last post I’ll make here about Mostly λazy. It has its own site/blog, so you should go subscribe there if you want to see more Clojure podcast news in the future.

Per usual, you, the Clojure community, have come through.  Ambrose Bonnaire-Sergeant will be attending the Conj and giving an introductory talk on logic programming, only made possible after $3,000 was raised through donations solicited via this blog and various pestering on Twitter, IRC, and the Clojure mailing list.

(As was mentioned from the start, the Conj organizers do provide a travel stipend, even a generous one for international speakers, but the costs of getting to North Carolina from Western Australia are far beyond that budget.  Thus the need for a fundraiser.)

Finally, I will soon be notifying the winners of the free copies of Clojure Programming from O’Reilly (signed by myself and hopefully my coauthors as well, Brian Carper and Christophe Grand) privately by email.  Your extraordinary generosity is a big part of how this effort was able to succeed.

Thank you!

To those that donated, thank you very, very much.  Your assistance is not taken for granted:

• Anonymous x 3
• Sam Aaron
• Harold Ancell
• Craig Andera
• Justin Balthrop
• Carlos Barboza
• James Christopher Bare
• Meikel Brandmeyer
• David Cabana
• Brian Cooley
• Aaron Crow
• Paul deGrandis
• Manuel Angel Rivas Diaz
• Alan Dipert
• Cameron Dorrat
• Kevin Downey
• Jim Duey
• Holger Durer
• Antoni Batchelli Estrada
• John Fingerhut
• Louis Franco
• Joseph Gallo
• Baishampayan Ghose
• Stephen Gilardi
• Juan Manuel Gimeno
• Christophe Grand
• Anthony Grimes
• Philip Hagelberg
• Mark Henderson
• Richard Hickey
• William Hidden
• Daniel Jomphe
• Geoffrey Knauth
• Vinod Kurup
• David Liebke
• Kent Mabee
• Christopher Maier
• Durgesh Mankekar
• Roberto Mannai
• Roberto Mannai
• Nathan Marz
• Carin Meier
• Dave Newton
• David Nolen
• Mark Nutter
• Alexey Ott
• Donald Parish
• Laurent Petit
• Michael Ravits
• Volker Schlecht
• Ryan Senior
• Patrick Shields
• Stuart Sierra
• Luo Tian
• Benny Tsai
• Moritz Ulrich
• Craig Worrall
• Koji Yusa

That said, the Clojure REPL is far from perfect.  A long time ago, I banged out this page describing what my ideal Clojure development environment would look like.  Much of what’s there is still an accurate reflection of my ideal end-state, and much of it is now available in Counterclockwise.  But, some things remain unfulfilled.

Like, when you are working with image data.  I do, all the time.  I’m parsing documents, producing rasterizations of them, performing all sorts of transformations and analysis on them, often in the REPL. Yet, I end up needing to have other tools around to do the simplest of things…like, oh, to look at an image.  Other fancier things should be possible, but let’s start with that and see where we end up.

This corresponds with this bullet in the aforementioned description of an ideal Clojure environment:

• Configurable pretty-printing of output
  • This also means being able to “print” non-textual data, such as having images drawn inline into the REPL if so desired, etc.
  • See Factor’s REPL for an example

So, I implemented opt-in “rich content” Clojure REPL interactions in nREPL (a tool-agnostic network Clojure REPL server & client implementation), with a first spike of client support for such “rich content” in Counterclockwise.  The result is something that I think can be the basis for breaking a lot of new ground in how we use the [Clojure] REPL.

Video demo follows, with some discussion of the implementation and ways clients can add their own response types and such dynamically (direct link for HD video):

The code for this is currently on nREPL HEAD (the SNAPSHOT coordinates are [org.clojure/tools.nrepl "0.0.6-SNAPSHOT"]), and this branch of Counterclockwise (which, along with a couple of refinements, will hopefully be rolled into the next release). It’s all bleeding-edge stuff, so I’m sure there’ll be lots of change to come, but it’s a start.

Go poke and prod at my approach, let me know what you think — and, hey, it’d be awesome if someone knocked out rich content REPL  support for another tool, be it jark (which already uses nREPL), vimclojure, Enclojure (both of which are planned to use nREPL eventually AFAIK?), or others.

(require '[cemerick.rummage :as sdb])
(def config (sdb/create-client "aws id" "aws secret-key"))
(put-attrs config "demo"
  {::sdb/id "foo" :name "value" :key #{50 60 65}})

The other is where the configuration is defined implicitly, usually using binding and dynamic scope (and sometimes via a set!-style setter for REPL usage convenience); here, in Clutch, accessing CouchDB:

(require '[com.ashafa.clutch :as clutch])
(clutch/with-db "localhost"
  (clutch/put-document {:a 5 :b 6} :id "foo"))

The latter is arguably more common, especially in database libraries; in addition to Clutch, you can see the dynamic pattern in play in java.jdbc‘s with-connection and congomongo‘s with-mongo.

From the perspective of a user (they are us!), I sometimes prefer dynamic scope to avoid verbosity, yet I often like to be explicit about configuration (and therefore, usually the target of my code’s activity) at other times, especially when dynamic scope isn’t appropriate or downright dangerous.  My preferences vacillate depending on what I’m doing, where I’m doing it, and what tools I’m using.  In any case, each library that requires configuration almost always requires that you work with it the way its author intended, so I am left with no joy half the time.

As an author of and contributor to such libraries (including the two mentioned above), perhaps I’m in a position to resolve this dilemma.

Irreconcilable differences

Consider any function that needs to make use of configuration, be it a session, a database connection, etc.  As we’ve seen, there seem to be only two implementation strategies: either take the configuration as an explicit argument, or assume the configuration has been bound dynamically elsewhere (as e.g. with-db does in the Clutch example above):

(defn explicit-save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

(def *config* nil)
(defn dynamic-save
  [data]
  ...do something with `data` in/at/with thing described by `*config*`...)

There is no way to unifying these two idioms. The only option is to manually provide additional arities of every single function in my API, delegating as necessary when it is suspected that dynamic scope is being used:

(def *config* nil)
(defn broken-save
  ([data] (broken-save *config* data))
  ([config data]
    ...do something with `data` in/at/with thing described by `config`...)))

In my opinion, this is a no-go from an implementor’s perspective: each additional library function implies extra maintenance for each additional arity, or the prospect of rigging up an alternative defn macro that adds the additional arity “automatically”…which doesn’t work if the API is to support rest or keyword args anywhere.

Not one to be disheartened, I’ve been working on an alternative that is either a proper solution, or a hack potentially even more ill-conceived than said defn macro.

Tasty brew: ns-publics + binding + partial

First, let’s implement our library as simply as possible, which means working with explicit arguments everywhere (you can build dynamic scope on top of simple functions, but it’s damn hard to make functions that depend on dynamic scope appear to do otherwise):

(defn save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

Now, let’s think about what *config* really represents in prior examples: it’s an implicit indication of the scope of an operation.  We can get a similar effect using partial, which returns a new function that will use the provided arguments as the first arguments to the original function; using it, we can call (a derivative of) our save function with a single argument (our data), making the configuration “implicit” again:

((partial save {:some :configuration})
 {:some :data})

That’s hardly a syntactic improvement over explicitly passing our configuration value explicitly.  However, what if we had a with-config macro that performed this partial evaluation for us, supplying the configuration value to each of our library’s functions so that, within the with-config macro’s scope, each of those functions could be called sans configuration?  Well, we have macros and a reified dynamic environment, so let’s have at it:

(def public-api (vals (ns-publics *ns*)))

(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'your.library.ns/public-api]
                           [var (partial @var config)]))
     ~@body))

Explanation is surely in order.  First, we need define our public API; this is just a seq of the public vars in our library’s namespace (which need to be dynamic since we’re going to be rebinding all of them; make sure to use ^:dynamic metadata on them if you’re using Clojure 1.3.0+).

(It seems sane to me that this seq should be filtered based on other metadata to ensure that only those functions that take configuration as their first argument are included.  An example of this is below.)

Second, all our with-config macro does is set up a dynamic scope, binding to each of our library’s vars new functions with the provided configuration partially applied.  Within that scope, we can omit any further reference to the configuration value, even though the foundational implementations of our library’s functions require explicit configuration.

Here’s a complete example (which requires Clojure 1.3.0 because of the ^: metadata notation — porting to Clojure 1.2.0 is simple enough, and left as an exercise):

(ns example)
(defn ^:api ^:dynamic save
  [config data]
  (println (format "Saved %s with %s" data config)))

(def public-api (->> (ns-publics *ns*)
                  vals
                  (filter (comp :api meta))
                  doall))
(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'example/public-api]
                             [var (partial @var ~config)]))
     ~@body))

The save function takes configuration explicitly; also, I’ve added ^:api to its var’s metadata so our public-api seq of vars can be filtered of vars that shouldn’t be affected by with-config‘s dynamic scope.  Now our library can support both explicit and dynamic specification of configuration, yet we never really thought at all about the dynamic case when implementing the library:

=> (save {:some :config} {:a 5 :b 6})
Saved {:a 5, :b 6} with {:some :config}
nil
=> (with-config {:some :config}
     (save {:a 5 :b 6}))
Saved {:a 5, :b 6} with {:some :config}
nil

Fin?

I love the flexibility this approach affords the user (usually me!), with, in relative terms, minor effort on the part of the library author.  I’m enough of a fan of it that I’m using it in Clutch (hopefully to be released soon as part of v0.3.0).

However, I should say that I’m not yet entirely at ease:

1. If your implementation of one public API function calls another, and the vars of both are being rebound by with-config(or its equivalent), then that intra-library function call is going to route through the var and get the function that already has the configuration value partially applied.My solution to this at the moment is to (ack!) use a defn-wrapping macro to define each public function in Clutch that pushes each definition into a closure containing all of the already-defined functions.  This keeps intra-library calls from getting mixed up in the dynamic scope that Clutch’s version of with-config might set up.
2. Except for the results from one (likely errant) REPL session, I believe that self-calls never route back through the var named in function position.  However, if there is any case where that’s not true (i.e. if self-calls do route through the named var), then self-calls would have the same problem (but not the same — or any — solution) as (1).
3. Using something like with-config, you’re looking at N partial invocations, N function instantiations, and N dynamic bindings for your library with N public functions, versus 0, 0, and 1 for dynamic configuration APIs that bind a single *config* var.  Insofar as all of the libraries that use this pattern that I know of are database and/or IO-related, this “overhead” can likely be discounted.  In any case, if you don’t want any overhead, with-config gives you the option of no dynamic scope at all.

If you have any comments, warnings, or rants about how this is evil, do share.

The result is clutch-clojurescript, which naturally builds on top of the Clutch library that I’ve collaborated on with Tunde Ashafa for some time now.

My motivations for doing this were manifold:

1. I quite enjoy using CouchDB, as its model and general philosophy meshes very naturally with my (and my tools’) disposition and the data I work with most often.
2. The operational hassle associated with maintaining a Clojure view server (which Clutch provides) configuration alongside my CouchDB installs was always a hassle.
3. I’ve been wanting to do more and more with Cloudant, but a Clojure view server is just not an option with a hosted database-as-a-service like that.
4. I can’t stand writing JavaScript.  Give me the reach of JavaScript, but with sane abstractions, homoiconicity (macros!), and data structures that aren’t braindead? Sign me up.

Feel free to go check out clutch-clojurescript: beat on it some, and let me know if it breaks on you.  I would eventually like to fold it into Clutch proper.  Beware some limitations though — repeated here from the README in part to draw attention to them:

• ClojureScript is not yet available as a proper library. This forces me to include some binary version of it in this git repo (a hefty 8.3MB!…which includes various google JavaScript UI bits that I’d hope would be broken out eventually), and bundle the necessary bits into the clutch-clojurescript jar. I would very much like to roll clutch-clojurescript’s functionality into Clutch proper, but I’ll not do so until the latter can rely upon a ClojureScript dependency.
• ClojureScript / Google Closure produces a very large code footprint, even for the simplest of view functions. This is apparently an item of active development in ClojureScript.

  In any case, the code size of a view function string should have little to no impact on runtime performance of that view. The only penalty to be paid should be in view server initialization, which should be relatively infrequent. Further, the vast majority of view runtime is dominated by IO and actual document processing, not the loading of a handful of JavaScript functions.

• To my surprise (and shock/horror), the version of Spidermonkey that is used by CouchDB (and Couchbase Single, and Cloudant) does not treat regular expression literals properly — they work fine as arguments, e.g. string.match(/foo/), but e.g. /foo/.exec("string") fails.  Using the RegExp() function with a string argument *does* work.  This was reported a long time ago, but has had little attention (though I’m trying to stir it up a bit).

  I’m hoping to get to the bottom of this sooner or later, but I wonder if it’d be worthwhile to change the ClojureScript reader to emit (js/RegExp "foo") calls instead of /foo/ literals (and hope that gClosure doesn’t optimize the former into the latter)?  After all, there’s lots of CouchDB deployments out there with apparently broken spidermonkey installs/configurations, and likely lots of other apps/servers/environments in similarly dire straits.

Finally, here are the slides from my talk at the BACUG (download/view PDF):

Update 2011-11-08T01:41  We did it! Details and acknowledgements for all who donated now here!

About a year ago, I had the distinct privilege of having a front-row seat when the generosity of the Clojure community (and others!) helped Anthony Grimes (aka Raynes) attend the 2010 Clojure Conj.  Witnessing the success of that process made me certain that I would attempt to coordinate another “scholarship” fundraising drive for this year’s Clojure Conj.

Like Anthony last year, I wanted to help someone that:

1. had already contributed significantly to the Clojure community,
2. clearly has great potential, but
3. without some financial assistance, would not be able to attend the Conj and benefit from the meeting-of-the-minds that such conferences uniquely enable.

Over the summer, it became clear to me that Ambrose Bonnaire-Sergeant was that someone.  A computer science undergraduate at the University of Western Australia in Perth, Ambrose has distinguished himself in the Clojure community this year.

His introduction to many was PragPub magazine publishing his Growing a DSL with Clojure article in July.  In the article, Ambrose led us through the building of a simplified shell-scripting DSL in Clojure modeled on the open source stevedore library that he’s contributed to.  More recently, Ambrose has been busy hacking away at significant contributions to the core.logic (logic programming) and core.match (optimized pattern matching and predicate dispatch) libraries led by David Nolen, producing content like his Logic-Starter documentation project that provides a compelling logic programming primer using core.logic as the raw materials, and helping out with jark tooling.  Finally, his has been a consistently helpful voice on the main Clojure mailing list.

Now, getting Ambrose to the Conj is going to be somewhat more challenging than last year’s effort: he’s in Australia after all, which makes both the travel and lodging expenses quite steep.  To make matters even more interesting, flights from Perth to Raleigh/Durham seem to fluctuate in price significantly — from $2,500 to $3,500, depending on when and where I check them.

Thankfully, Ambrose did submit a talk proposal to the Conj — an Introduction to Logic Programming with Clojure — which was accepted!  This means that the Conj organizers will be picking up his lodging during the conference, as well as generously providing $1,000 to help defray Ambrose’s travel expenses.  Thus, we (only?) need to raise enough to cover the remainder of his flight, as well as two extra hotel nights (so he can recuperate from jet lag enough to deliver his talk without experiencing a sudden bout of narcolepsy!).

To be safe, I am therefore setting the fundraising goal to $3,000.  As long as the gods of airline pricing are not against us, this will safely cover the remainder of Ambrose’s expenses, allow him to attend the Conj, and allow you to enjoy his sure-to-be-enlightening introductory logic programming talk.  Even moreso than last year, it’s a sizable sum; but, again, I think this is a worthy cause, and I suspect the Clojure community will rise to meet the challenge.

(If any funds remain, they will be used to provide discounted admission to the Conj for other students.)

I’ve set up a PayPal donation button below, along with a progress bar.  Some FYIs:

• I’m just reusing Snowtide‘s PayPal account for this; so, you’ll be seeing its logo through the checkout process and “snowtide” on your credit card statement.
• I’ll be manually updating the progress bar, so don’t expect to see it go up immediately after donating!
• I’ll shut down the donation button once we hit the $3,000 target.
• I’ll add your name to this page as thanks, unless you notify me otherwise, either (preferably) in the comments associated with your PayPal transaction or by email/twitter/irc.

Finally, as an added incentive: at the end of the fundraiser, the three people with the largest donations will each receive a copy of my forthcoming Clojure book, Clojure Programming from O’Reilly, signed by me (and probably my co-conspirators as well, Christophe Grand and Brian Carper).

If you have any questions at all, leave a comment here, or don’t hesitate to contact me if it your question involves confidential details (payment issues, etc).

We did it!

Details and acknowledgements for all who donated now here!

Anyway, he delivers a great zinger at the end, a tangential response to a question about the aims of Semantic Web technologies as they relate to his work on Elephant:

When w3c decided to not use [s-expressions], but intead imitate SGML [for HTML], that showed a certain capacity to make mistakes — which, probably, they haven’t lost.

Nice.

Afterwards, OGINO Masanori contacted me, offering to produce a Japanese translation of the flowchart.  The result looks great…thank you, Masanori!  We both hope it is useful to all Japanese Clojure programmers:

holds more than 449 billion objects and processes up to 290,000 requests per second for them at peak times

I’m a very happy user of S3 and much of the rest of AWS, but seeing figures like this forces me into risk-assessment mode: at what point does S3 (or similar services) break?

Yes, S3 has had outages (2008 was a bad year), but these were fundamentally minor.  In the scheme of things, the equivalent of someone tripping over the rack’s power cord.

What I’m wondering about is, where does S3 et al. start hitting fundamental limits that amp problems up from oops to ohhh shit?

S3 is presumably the largest system of its kind ever; it’s not clear to me that anyone would really know what its failure modes, thresholds, or weak links might be as it continues to grow. Anything from a breakdown in the Dynamo architecture to hard infrastructure limits to failures in operations and management strike me as plausible. What will we see first: data loss, increases in latency, repeated catastrophic outages, or “other”?