A refresh of Clojure Atlas

I’m sorry to admit that I let the Clojure Atlas wilt a bit over the past year or so. (I was a little busy!)  However, I am conversely quite happy to say that that’s over now; Clojure Atlas has been refreshed to add editions for Clojure v1.3.0 and v1.4.0.

(If you don’t know what Clojure Atlas is, head on over and check out the snazzy new demo/tour video.)

Other highlights include:

Pricing changes

I think the previous pricing was too high.  (You never know until you try.)  Pricing has been lowered, and I’ve added a fun option whereby you can get any edition of Clojure Atlas for just $5.  I don’t quite know what I’ll end up doing for upgrades going forward, but you will definitely be able to stay current without paying the full boat each time.

Free upgrades

Between the too-high pricing and the far-too-long period between the initial release of Clojure Atlas and now, those that prepaid for access to the Clojure v1.3.0 Atlas how have access to all of them, up to and including v1.4.0.  Those early significant supporters will also get free upgrades to all future Clojure Atlas revisions.  Thanks, guys and gals.

If you only purchased the Atlas for Clojure v1.2.0 previously, your account has been upgraded to include the Atlas for v1.3.0.

Ontology improvements

Aside from the obvious additions that needed to go in to reflect changes in Clojure v1.3.0 and v1.4.0, the ontology has been improved significantly to be more comprehensive and more accurate.  In addition, I’ve started adding detailed documentation (for example) to subjects/nodes within the ontology that I’ve added (in contrast to vars, which in general already have documentation of their own).

Visualization improvements

The graph visualization is certainly far from perfect, but I’ve tweaked it a fair bit to get it to “settle” faster than it did before.  I’m also pondering a complete reworking of the visualization to make it deterministic (rather than using a particle simulation as it does now).

No more PayPal

Many people balked at using PayPal — and believe me, no one is happier than I to be rid of it at this point.  Payments are now all handled courtesy of Stripe, which has been a dream to work with.

Friend: an extensible authentication and authorization library for Clojure Ring webapps and services

Say hello to my little Friend.

There’s plenty of technical stuff in the README to chew on if you like.  In short, I’m hoping this can eventually be a warden/spring-security/everyauth /omniauth for Clojure; that is, a common abstraction for authentication and authorization mechanisms.  Clojure has been around long enough that adding pedestrian things like form and HTTP Basic and $AUTH_METHOD_HERE to a Ring application should be easy.  Right now, it’s not: either you’re pasting together a bunch of different libraries that don’t necessarily compose well together, or you get drawn into shaving the authentication and authorization yaks for the fifth time in your life so you can sleep well at night.

Hopefully Friend will make this a solved problem, or at least push things in that direction.  It plays nice with all of the best principles of Ring, and includes support for:

  • form, HTTP Basic, and OpenID authentication
  • role-based authorization (optionally using hierarchical roles via Clojure’s derive and isa?)
  • su capabilities (multiple login support / a.k.a. “log in as”)
  • channel security (i.e. HTTPS-only for certain Ring routes)
  • …and more

Most importantly, it takes a stab at a couple of core abstractions for others to drop in other authentication workflows, e.g. OAuth in all of its incarnations, NTLM, BrowserID, etc. etc. etc.  There are already plenty of Clojure implementations for all sorts of authentication methods; hopefully someone (you?!) will step up and bring one of them to the party, so anyone’s Friend-empowered Clojure webapp can easily offer any or all of them with a minimum of suffering.

Finally: frankly, it’s absurd that I’m writing security-related stuffs.  (I know it hardly ever works out that way, but it seems like some experts somewhere should be taking care of this.)  It would be a great thing if you were to beat on Friend and try to find exploits, general breakage, etc., especially if you have prior experience in this area.

Originally posted on Mostly λazy…a Clojure podcast:

Recorded November 12th, 2011, the fourth and final recording in a series of conversations from Clojure Conj 2011.

Chris Houser (usually known as chouser online) has been working with Clojure longer than nearly anyone else; he started tinkering with the language in early 2008, and was a fixture in #clojure irc and on the mailing list for years.  His contributions to the language, early libraries, and community through his always genial and insightful presence are hard to overstate.  More recently, he has coauthored the excellent Joy of Clojure along with Michael Fogus, and is now working with Clojure daily over at Lonocloud.

It’s been my privilege to know and work with Chris a bit over the years, and, as always, it was great to talk with him in person.

Enjoy!

Listen:

Or, download the mp3 directly.

Discrete Topics

  • “Everything I learned, I [learned] on irc?!”
  • Macros…

View original 111 more words

‘Clojure Programming’ book finished

Yes — it’s finished! :-D

Early last month, after writing 190,000 words, editing away scads more, assembling and testing more than 1,000 code snippets and 20 full sample projects, and conceptualizing dozens of illustrations, Christophe, Brian, and I declared Clojure Programming done.  It’s been writhing its way through O’Reilly’s editorial process ever since.

I’d hoped that the book would be published before Clojure/West in mid-March, but alas, it was not to be.  It looks like it’ll drop in mid-April.

However, fret not! If you want to dig into Clojure Programming right away, you can read the final draft of it online.  Of course, you can preorder the dead-tree version of it as well; easy links to both options are available at clojurebook.com.  There, you’ll also find a full table of contents, some basic info on the book, and a way to join the clojurebook.com mailing list and a pointer to the book’s Twitter account.  We’ll be pushing various Clojure tips and links to useful tools and resources and announcing the availability of all sorts of book-related content on the site through the mailing list and Twitter feed; and, if things work out as I hope, some early access to and/or special offers for things that will help you get the most out of your Clojure experience in general.

So, thanks for your patience.  I think the book will end up being worth it.  Of course, I have to thank my coauthors; without Brian and Christophe, it simply would never have been finished, nor would it be as good as it is.  There’s a ton of other people that deserve credit too, but you’ll have to buy the book and read the acknowledgements to learn about them…

Originally posted on Mostly λazy…a Clojure podcast:

Recorded November 12th, 2011, third in a series of conversations from Clojure Conj 2011.

I caught up with Hugo Duncan and Antoni Batchelli (everyone calls him Toni ;-) during one of the lunch breaks at the Conj.  These guys have been on a tear with Pallet, an open source Clojure project that Hugo started in early 2010 to shave one of the hairiest yaks around, the automation of provisioning and management of computing infrastructure.  The result is a tool and library that provides a classically Clojure abstraction for controlling nearly any environment, from cloud nodes to virtual machines to the rackmounts you have downstairs.  Since it is their full-time job — Toni and Hugo have built a business around the project — most of our discussion centers on Pallet, its history, and how people are using it.

Near the end, Hugo and I talk some about his other…

View original 270 more words

Introducing: Mostly λazy (a Clojure podcast)

As some of you may know already, I’ve started a Clojure podcast called Mostly λazyThe first episode is in the can and has been published.  There are RSS feeds for your readers and podcatchers there for the taking, and I’ll be working on getting the feed into iTunes shortly.

The initial announcement post for Mostly λazy drew a lot of really good ideas and suggestions, many of which I’ll be stealing. :-D I’m really glad that a lot of people find the concept of a Clojure podcast appealing; whilst I’m fundamentally doing this to enjoy myself, it’s nice to know that there will be at least a dozen or so people listening… ;-)

I’ll be at Clojure Conj 2011 this week, and will likely capture a bunch of conversations with people there.  That should provide me with a decent backlog of material that I can trickle out through the rest of the year, perhaps mixed in with some shorter segments based on the aforementioned suggestions.  In any case, I hope anyone who does listen finds the experience entertaining and engaging.

By the way, this is likely the first and last post I’ll make here about Mostly λazy. It has its own site/blog, so you should go subscribe there if you want to see more Clojure podcast news in the future.

Ambrose has received his Clojure “scholarship”, thanks to you.

The eve of Clojure Conj 2011 is upon us, so I feel compelled get off my butt and finally close the loop on this one.

Per usual, you, the Clojure community, have come through.  Ambrose Bonnaire-Sergeant will be attending the Conj and giving an introductory talk on logic programming, only made possible after $3,000 was raised through donations solicited via this blog and various pestering on Twitter, IRC, and the Clojure mailing list.

(As was mentioned from the start, the Conj organizers do provide a travel stipend, even a generous one for international speakers, but the costs of getting to North Carolina from Western Australia are far beyond that budget.  Thus the need for a fundraiser.)

Finally, I will soon be notifying the winners of the free copies of Clojure Programming from O’Reilly (signed by myself and hopefully my coauthors as well, Brian Carper and Christophe Grand) privately by email.  Your extraordinary generosity is a big part of how this effort was able to succeed.

Thank you!

To those that donated, thank you very, very much.  Your assistance is not taken for granted:

  • Anonymous x 3
  • Sam Aaron
  • Harold Ancell
  • Craig Andera
  • Justin Balthrop
  • Carlos Barboza
  • James Christopher Bare
  • Meikel Brandmeyer
  • David Cabana
  • Brian Cooley
  • Aaron Crow
  • Paul deGrandis
  • Manuel Angel Rivas Diaz
  • Alan Dipert
  • Cameron Dorrat
  • Kevin Downey
  • Jim Duey
  • Holger Durer
  • Antoni Batchelli Estrada
  • John Fingerhut
  • Louis Franco
  • Joseph Gallo
  • Baishampayan Ghose
  • Stephen Gilardi
  • Juan Manuel Gimeno
  • Christophe Grand
  • Anthony Grimes
  • Philip Hagelberg
  • Mark Henderson
  • Richard Hickey
  • William Hidden
  • Daniel Jomphe
  • Geoffrey Knauth
  • Vinod Kurup
  • David Liebke
  • Kent Mabee
  • Christopher Maier
  • Durgesh Mankekar
  • Roberto Mannai
  • Roberto Mannai
  • Nathan Marz
  • Carin Meier
  • Dave Newton
  • David Nolen
  • Mark Nutter
  • Alexey Ott
  • Donald Parish
  • Laurent Petit
  • Michael Ravits
  • Volker Schlecht
  • Ryan Senior
  • Patrick Shields
  • Stuart Sierra
  • Luo Tian
  • Benny Tsai
  • Moritz Ulrich
  • Craig Worrall
  • Koji Yusa

Enabling rich(er) interactions in the Clojure REPL

I love the Clojure REPL.  I’ve never been more productive in any other environment.  In particular, I happen to like the REPL and general development experience provided by Counterclockwise, the Eclipse Clojure plugin.

That said, the Clojure REPL is far from perfect.  A long time ago, I banged out this page describing what my ideal Clojure development environment would look like.  Much of what’s there is still an accurate reflection of my ideal end-state, and much of it is now available in Counterclockwise.  But, some things remain unfulfilled.

Like, when you are working with image data.  I do, all the time.  I’m parsing documents, producing rasterizations of them, performing all sorts of transformations and analysis on them, often in the REPL. Yet, I end up needing to have other tools around to do the simplest of things…like, oh, to look at an image.  Other fancier things should be possible, but let’s start with that and see where we end up.

This corresponds with this bullet in the aforementioned description of an ideal Clojure environment:

  • Configurable pretty-printing of output
    • This also means being able to “print” non-textual data, such as having images drawn inline into the REPL if so desired, etc.
    • See Factor’s REPL for an example

So, I implemented opt-in “rich content” Clojure REPL interactions in nREPL (a tool-agnostic network Clojure REPL server & client implementation), with a first spike of client support for such “rich content” in Counterclockwise.  The result is something that I think can be the basis for breaking a lot of new ground in how we use the [Clojure] REPL.

Video demo follows, with some discussion of the implementation and ways clients can add their own response types and such dynamically (direct link for HD video):

The code for this is currently on nREPL HEAD (the SNAPSHOT coordinates are [org.clojure/tools.nrepl "0.0.6-SNAPSHOT"]), and this branch of Counterclockwise (which, along with a couple of refinements, will hopefully be rolled into the next release). It’s all bleeding-edge stuff, so I’m sure there’ll be lots of change to come, but it’s a start.

Go poke and prod at my approach, let me know what you think — and, hey, it’d be awesome if someone knocked out rich content REPL  support for another tool, be it jark (which already uses nREPL), vimclojure, Enclojure (both of which are planned to use nREPL eventually AFAIK?), or others.

À la carte configuration in Clojure APIs

There are two dominant configuration patterns in Clojure libraries. The first is where configuration is provided explicitly via the first argument; here, in Rummage, accessing Amazon’s SimpleDB:

(require '[cemerick.rummage :as sdb])
(def config (sdb/create-client "aws id" "aws secret-key"))
(put-attrs config "demo"
  {::sdb/id "foo" :name "value" :key #{50 60 65}})

The other is where the configuration is defined implicitly, usually using binding and dynamic scope (and sometimes via a set!-style setter for REPL usage convenience); here, in Clutch, accessing CouchDB:

(require '[com.ashafa.clutch :as clutch])
(clutch/with-db "localhost"
  (clutch/put-document {:a 5 :b 6} :id "foo"))

The latter is arguably more common, especially in database libraries; in addition to Clutch, you can see the dynamic pattern in play in java.jdbc‘s with-connection and congomongo‘s with-mongo.

From the perspective of a user (they are us!), I sometimes prefer dynamic scope to avoid verbosity, yet I often like to be explicit about configuration (and therefore, usually the target of my code’s activity) at other times, especially when dynamic scope isn’t appropriate or downright dangerous.  My preferences vacillate depending on what I’m doing, where I’m doing it, and what tools I’m using.  In any case, each library that requires configuration almost always requires that you work with it the way its author intended, so I am left with no joy half the time.

As an author of and contributor to such libraries (including the two mentioned above), perhaps I’m in a position to resolve this dilemma.

Irreconcilable differences

Consider any function that needs to make use of configuration, be it a session, a database connection, etc.  As we’ve seen, there seem to be only two implementation strategies: either take the configuration as an explicit argument, or assume the configuration has been bound dynamically elsewhere (as e.g. with-db does in the Clutch example above):

(defn explicit-save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

(def *config* nil)
(defn dynamic-save
  [data]
  ...do something with `data` in/at/with thing described by `*config*`...)

There is no way to unifying these two idioms. The only option is to manually provide additional arities of every single function in my API, delegating as necessary when it is suspected that dynamic scope is being used:

(def *config* nil)
(defn broken-save
  ([data] (broken-save *config* data))
  ([config data]
    ...do something with `data` in/at/with thing described by `config`...)))

In my opinion, this is a no-go from an implementor’s perspective: each additional library function implies extra maintenance for each additional arity, or the prospect of rigging up an alternative defn macro that adds the additional arity “automatically”…which doesn’t work if the API is to support rest or keyword args anywhere.

Not one to be disheartened, I’ve been working on an alternative that is either a proper solution, or a hack potentially even more ill-conceived than said defn macro.

Tasty brew: ns-publics + binding + partial

First, let’s implement our library as simply as possible, which means working with explicit arguments everywhere (you can build dynamic scope on top of simple functions, but it’s damn hard to make functions that depend on dynamic scope appear to do otherwise):

(defn save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

Now, let’s think about what *config* really represents in prior examples: it’s an implicit indication of the scope of an operation.  We can get a similar effect using partial, which returns a new function that will use the provided arguments as the first arguments to the original function; using it, we can call (a derivative of) our save function with a single argument (our data), making the configuration “implicit” again:

((partial save {:some :configuration})
 {:some :data})

That’s hardly a syntactic improvement over explicitly passing our configuration value explicitly.  However, what if we had a with-config macro that performed this partial evaluation for us, supplying the configuration value to each of our library’s functions so that, within the with-config macro’s scope, each of those functions could be called sans configuration?  Well, we have macros and a reified dynamic environment, so let’s have at it:

(def public-api (vals (ns-publics *ns*)))

(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'your.library.ns/public-api]
                           [var (partial @var config)]))
     ~@body))

Explanation is surely in order.  First, we need define our public API; this is just a seq of the public vars in our library’s namespace (which need to be dynamic since we’re going to be rebinding all of them; make sure to use ^:dynamic metadata on them if you’re using Clojure 1.3.0+).

(It seems sane to me that this seq should be filtered based on other metadata to ensure that only those functions that take configuration as their first argument are included.  An example of this is below.)

Second, all our with-config macro does is set up a dynamic scope, binding to each of our library’s vars new functions with the provided configuration partially applied.  Within that scope, we can omit any further reference to the configuration value, even though the foundational implementations of our library’s functions require explicit configuration.

Here’s a complete example (which requires Clojure 1.3.0 because of the ^: metadata notation — porting to Clojure 1.2.0 is simple enough, and left as an exercise):

(ns example)
(defn ^:api ^:dynamic save
  [config data]
  (println (format "Saved %s with %s" data config)))

(def public-api (->> (ns-publics *ns*)
                  vals
                  (filter (comp :api meta))
                  doall))
(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'example/public-api]
                             [var (partial @var ~config)]))
     ~@body))

The save function takes configuration explicitly; also, I’ve added ^:api to its var’s metadata so our public-api seq of vars can be filtered of vars that shouldn’t be affected by with-config‘s dynamic scope.  Now our library can support both explicit and dynamic specification of configuration, yet we never really thought at all about the dynamic case when implementing the library:

=> (save {:some :config} {:a 5 :b 6})
Saved {:a 5, :b 6} with {:some :config}
nil
=> (with-config {:some :config}
     (save {:a 5 :b 6}))
Saved {:a 5, :b 6} with {:some :config}
nil

Fin?

I love the flexibility this approach affords the user (usually me!), with, in relative terms, minor effort on the part of the library author.  I’m enough of a fan of it that I’m using it in Clutch (hopefully to be released soon as part of v0.3.0).

However, I should say that I’m not yet entirely at ease:

  1. If your implementation of one public API function calls another, and the vars of both are being rebound by with-config(or its equivalent), then that intra-library function call is going to route through the var and get the function that already has the configuration value partially applied.My solution to this at the moment is to (ack!) use a defn-wrapping macro to define each public function in Clutch that pushes each definition into a closure containing all of the already-defined functions.  This keeps intra-library calls from getting mixed up in the dynamic scope that Clutch’s version of with-config might set up.
  2. Except for the results from one (likely errant) REPL session, I believe that self-calls never route back through the var named in function position.  However, if there is any case where that’s not true (i.e. if self-calls do route through the named var), then self-calls would have the same problem (but not the same — or any — solution) as (1).
  3. Using something like with-config, you’re looking at N partial invocations, N function instantiations, and N dynamic bindings for your library with N public functions, versus 0, 0, and 1 for dynamic configuration APIs that bind a single *config* var.  Insofar as all of the libraries that use this pattern that I know of are database and/or IO-related, this “overhead” can likely be discounted.  In any case, if you don’t want any overhead, with-config gives you the option of no dynamic scope at all.

If you have any comments, warnings, or rants about how this is evil, do share.

Writing CouchDB Views using ClojureScript

UPDATE [2012-05-07]: clutch-clojurescript is now deprecated, as its functionality has been rolled into Clutch proper in toto.  Documentation for the feature can be found here.

While I was in San Fransisco for JavaOne, I was lucky enough to be invited to speak at the Bay Area Clojure User Group (thanks, Sean and Toni!).  It was a great time, and gave me the kick in the pants I needed to finish hacking away at my first project involving ClojureScript: using it to write view functions for CouchDB.

The result is clutch-clojurescript, which naturally builds on top of the Clutch library that I’ve collaborated on with Tunde Ashafa for some time now.

My motivations for doing this were manifold:

  1. I quite enjoy using CouchDB, as its model and general philosophy meshes very naturally with my (and my tools’) disposition and the data I work with most often.
  2. The operational hassle associated with maintaining a Clojure view server (which Clutch provides) configuration alongside my CouchDB installs was always a hassle.
  3. I’ve been wanting to do more and more with Cloudant, but a Clojure view server is just not an option with a hosted database-as-a-service like that.
  4. I can’t stand writing JavaScript.  Give me the reach of JavaScript, but with sane abstractions, homoiconicity (macros!), and data structures that aren’t braindead? Sign me up.

Feel free to go check out clutch-clojurescript: beat on it some, and let me know if it breaks on you.  I would eventually like to fold it into Clutch proper.  Beware some limitations though — repeated here from the README in part to draw attention to them:

  • ClojureScript is not yet available as a proper library. This forces me to include some binary version of it in this git repo (a hefty 8.3MB!…which includes various google JavaScript UI bits that I’d hope would be broken out eventually), and bundle the necessary bits into the clutch-clojurescript jar. I would very much like to roll clutch-clojurescript’s functionality into Clutch proper, but I’ll not do so until the latter can rely upon a ClojureScript dependency.
  • ClojureScript / Google Closure produces a very large code footprint, even for the simplest of view functions. This is apparently an item of active development in ClojureScript.In any case, the code size of a view function string should have little to no impact on runtime performance of that view. The only penalty to be paid should be in view server initialization, which should be relatively infrequent. Further, the vast majority of view runtime is dominated by IO and actual document processing, not the loading of a handful of JavaScript functions.
  • To my surprise (and shock/horror), the version of Spidermonkey that is used by CouchDB (and Couchbase Single, and Cloudant) does not treat regular expression literals properly — they work fine as arguments, e.g. string.match(/foo/), but e.g. /foo/.exec("string") fails.  Using the RegExp() function with a string argument *does* work.  This was reported a long time ago, but has had little attention (though I’m trying to stir it up a bit).I’m hoping to get to the bottom of this sooner or later, but I wonder if it’d be worthwhile to change the ClojureScript reader to emit (js/RegExp "foo") calls instead of /foo/ literals (and hope that gClosure doesn’t optimize the former into the latter)?  After all, there’s lots of CouchDB deployments out there with apparently broken spidermonkey installs/configurations, and likely lots of other apps/servers/environments in similarly dire straits.

Finally, here are the slides from my talk at the BACUG (download/view PDF):