Jay Fields's Blog, page 11

Clojure: Converting a Java Object to a Clojure Map

Clojure has a great library that helps facilitate most things you'd want to do, but it's also easy to roll your own solution if you need something more specific. This blog entry is about converting an object to a map, something Clojure already has a function for: bean. From the docs:

Takes a Java object and returns a read-only implementation of the
map abstraction based upon its JavaBean properties.

The bean function is easy enough to use, as the following example shows (formatted for readability).user=> (import 'java.util.Calendar)
java.util.Calendar
user=> (-> (Calendar/getInstance) .getTime bean)
{:seconds 16,
:date 2,
:class java.util.Date,
:minutes 35,
:hours 12,
:year 111,
:timezoneOffset 240,
:month 4,
:day 1,
:time 1304354116038}The bean function is simple and solves 99% of the problems I encounter. Generally I'm destructuring a value that I pull from a map that was just created from an object; bean works perfectly for this.

However, I have run across two cases where I needed something more specialized.

The first case involved processing a large amount of data in a very timely fashion. We found that garbage collection pauses were slowing us down too much, and beaning the objects that were coming in was causing a large portion of the garbage. Since we were only dealing with one type of object we ended up writing something specific that only grabbed the fields we cared about.

The second case involved working with less friendly Java objects that needed to be converted to maps that were going to be used many times throughout the system. In general it isn't a big deal working with ugly Java objects. You can destructure the map into nice var names, or you can use the
The following example shows the definition of obj->map and the REPL output of using it.user=> (import 'java.util.Date)
java.util.Date
user=> (defmacro obj->map [o & bindings]
(let [s (gensym "local")]
`(let [~s ~o]
~(->> (partition 2 bindings)
(map (fn [[k v]]
(if (vector? v)
[k (list (last v) (list (first v) s))]
[k (list v s)])))
(into {})))))
#'user/obj->map
user=> (obj->map (Date. 2012 1 31)
:month .getMonth
:year [.getYear #(* 2 %)]
:day .getDate)
{:month 2, :year 4024, :day 2}As you can see from the output, the macro grabs the values of the methods you specify and stores them in a map. At one point I also needed to do a bit of massaging of a value, so I included the ability to use a vector to specify the method you care about and a function to apply to the value (the result of the function will become the value). In the example, you can see that the :year is double the value of what .getYear returns.

In general you should stick with bean, but if you need something more discriminating, obj->map might work for you.
© Jay Fields - www.jayfields.com

Clojure: State Management

Those unfamiliar with Clojure are often interested in how you manage changing state within your applications. If you've heard a few things about Clojure but haven't really looked at it, I wouldn't be surprised if you thought it was impossible to write a "real" application with Clojure since "everything is immutable". I've even heard a developer that I respect make the mistake of saying: we're not going to use Clojure because it doesn't handle state well.

Clearly, state management in Clojure is greatly misunderstood.

I actually had a hard time not calling this blog entry "Clojure, it's about state". I think state shapes Clojure more than any other influence; it's the core of the language (as far as I can tell). Rich Hickey has clearly spent a lot of time thinking about state - there's an essay at http://clojure.org/state which describes common problems with a traditional approach to state management and Clojure's solutions.

Rich's essay does a good job of succinctly discussing his views on state; you should read it before you continue with this entry. The remainder of this entry will give examples of how you can manage state using Clojure's functions.

At the end of Rich's essay he says:

In the local case, since Clojure does not have mutable local variables, instead of building up values in a mutating loop, you can instead do it functionally with recur or reduce.

Before we get to reduce, let's start with the simplest example. You have an array of ints and you want to double each integer. In a language with mutable state you can loop through the array and build a new array with each integer doubled.for (int i=0; i < nums.length; i++) {
result.add(nums[i] * 2);
}In Clojure you would build the new array by calling the map function with a function that doubles each value. (I'm using Clojure 1.2)
user=> (map (fn [i] (* i 2)) [1 2 3])
(2 4 6)If you're new to Clojure there's a few things worth mentioning. "user=>" is a REPL prompt. You enter some text and hit enter and the text is evaluated. If you've completed the list (closed the parenthesis), the results of evaluating that list will be printed to the following line.

I remember what I thought the first time I looked at a lisp, and I know the code might not look like readable code, so here's a version that breaks up a few of the concepts and might make it easier to digest the example.user=> (defn double-int [i] (* i 2))
#'user/double-int
user=> (def the-array [1 2 3])
#'user/the-array
user=> (map double-int the-array)
(2 4 6)In the first Clojure example you call the fn function to create an anonymous function, that was then passed to the map function (to be applied to each element of the array). The map function is a high order function that can take an anonymous function (example 1) or a named function (double-int, example 2). In Clojure (def ...) is a special form that allows you to define a var and defn is a function that allows you to easily define a function and assign it to a var. The syntax for defn is pretty straightforward, the first argument is the name, the second argument is the argument list of the new function, and any additional forms are the body of the function you are defining.

Once you get used to Clojure's syntax you can even have a bit of fun with your function naming that might result in concise and maintainable code.user=> (defn *2 [i] (* 2 i))
#'user/*2
user=> (map *2 [1 2 3])
(2 4 6)but, I digress.

Similarly, you may want to sum the numbers from an array.for (int i = 0; i < nums.length; i++) {
result += nums[i];
}You can achieve goal of reducing an array to a single value in Clojure using the reduce function.user=> (reduce + [1 2 3])
6Clojure has several functions that allow you to create new values from existing values, which should be enough to solve any problem where you would traditionally use local mutable variables.

For non-local mutable state you generally have 3 options: atoms, refs, and agents.

When I started programming in Clojure, atoms were my primary choice for mutable state. Atoms are very easy to use and only require that you know a few functions to interact with them. Let's assume we're building a trading application that needs to keep around the current price of Apple. Our application will call our apple-price-update function when a new price is received and we'll need to keep that price around for (possible) later usage. The example below shows how you can use an atom to track the current price of Apple.user=> (def apple-price (atom nil))
#'user/apple-price
user=> (defn update-apple-price [new-price] (reset! apple-price new-price))
#'user/update-apple-price
user=> @apple-price
nil
user=> (update-apple-price 300.00)
300.0
user=> @apple-price
300.0
user=> (update-apple-price 301.00)
301.0
user=> (update-apple-price 302.00)
302.0
user=> @apple-price
302.0The above example demonstrates how you can create a new atom and reset its value with each price update. The reset! function sets the value of the atom synchronously and returns its new value. You can also query the price of apple at any time using @ (or deref).

If you're coming from a Java background the example above should be the easiest to relate to. Each time we call the update-apple-price function our state is set to a new value. However, atoms provide much more value than simply being a variable that you can reset.

You may remember the following example from Java Concurrency in Practice.@NotThreadSafe
public class UnsafeSequence {
private int value;

/**
* Returns a unique value.
*/
public int getNext() {
return value++;
}
}The book explains why this could cause potential problems.

The problem with UnsafeSequence is that with some unlucky timing, two threads could call getNext and receive the same value. The increment notation, nextValue++, may appear to be a single operation, but is in fact three separate operations: read the value, add one to it, and write out the new value. Since operations in multiple threads may be arbitrarily interleaved by the runtime, it is possible for two threads to read the value at the same time, both see the same value, and then both add one to it. The result is that the same sequence number is returned from multiple calls in different threads.

We could write a get-next function using a Clojure atom and the same race condition would not be a concern.user=> (def uniq-id (atom 0))
#'user/uniq-id
user=> (defn get-next [] (swap! uniq-id inc))
#'user/get-next
user=> (get-next)
1
user=> (get-next)
2The above code demonstrates the result of calling get-next multiple times (the inc function just adds one to the value passed in). Since we aren't in a multithreaded environment the example isn't exactly breathtaking; however, what's actually happening under the covers is described very well on clojure.org/atoms -

[Y]ou change the value by applying a function to the old value. This is done in an atomic manner by swap! Internally, swap! reads the current value, applies the function to it, and attempts to compare-and-set it in. Since another thread may have changed the value in the intervening time, it may have to retry, and does so in a spin loop. The net effect is that the value will always be the result of the application of the supplied function to a current value, atomically.

Also, remember that changes to atoms are synchronous, so our get-next function will never return the same value twice.

(note: while Java already provides an AtomicInteger class for handling this issue - that's not the point. The point of the example is to show that an Atom is safe to use across threads.)

If you're truly interested in verifying that an atom is safe across threads, The Joy of Clojure provides the following snippet of code (as well as a wonderful explanation of all things Clojure, including mutability).(import '(java.util.concurrent Executors))

(def *pool* (Executors/newFixedThreadPool
(+ 2 (.availableProcessors (Runtime/getRuntime)))))

(defn dothreads [f & {thread-count :threads exec-count :times :or {thread-count 1 exec-count 1}}]
(dotimes [t thread-count]
(.submit *pool* #(dotimes [_ exec-count] (f)))))

(def ticks (atom 0))
(defn tick [] (swap! ticks inc))
(dothreads tick :threads 1000 :times 100)
@ticks ;=> 100000There you have it, 1000 threads updated ticks 100 times without issue.

Atoms work wonderfully when you want to insure atomic updates to an individual piece of state; however, it probably wont be long before you find yourself wanting to coordinate some type of state update. For example, if you're running an online store, when a customer cancels an order the order is either active or cancelled; however, the order should never be active and cancelled. If you were to keep a set of active orders and a set of cancelled orders, you would never want to have an order be in both sets at the same time. Clojure addresses this issue by using refs. Refs are similar to atoms, but they also participate in coordinated updates.

The following example shows the cancel-order function moving an order-id from the active orders set into the cancelled orders set.user=> (def active-orders (ref #{2 3 4}))
#'user/active-orders
user=> (def cancelled-orders (ref #{1}))
#'user/cancelled-orders
user=> (defn cancel-order [id]
(dosync
(commute active-orders disj id)
(commute cancelled-orders conj id)))
#'user/cancel-order
user=> (cancel-order 2)
#{1 2}
user=> @active-orders
#{3 4}
user=> @cancelled-orders
#{1 2}As you can see from the example, we're moving an order id from active to cancelled. Again, our REPL session doesn't show the power of what's going on with a ref, but clojure.org/refs contains a good explanation -

All changes made to Refs during a transaction (via ref-set, alter or commute) will appear to occur at a single point in the 'Ref world' timeline (its 'write point').

The above quote is actually only 1 item in a 10 point list that discusses what's actually going on. It's worth reviewing the list a few times until you feel comfortable with everything that's going on. But, you don't need to completely understand everything to get started. You can begin to experiment with refs anytime you know you need coordinated changes to more than one piece of state.

When you first begin to look at refs you may wonder if you should use commute or alter. For most cases commute will provide more concurrency and is preferred; however, you may need to guarantee that the ref has not been updated during the life of the current transaction. This is generally the case where alter comes into play. The following example shows using commute to update two values. The example demonstrates that the pairs are always updated only once; however, it also shows that the function is simply applied to the current value, so the incrementing is not sequential and @uid can be dereferenced to the same value multiple times.user=> (def uid (ref 0))
#'user/uid
user=> (def used-id (ref []))
#'user/used-id
user=> (defn use-id []
(dosync
(commute uid inc)
(commute used-id conj @uid)))
#'user/use-id
user=> (dothreads use-id :threads 10 :times 10)
nil
user=> @used-id
[1 2 3 4 5 6 7 8 9 10 ... 89 92 92 94 93 94 97 97 99 100]The above example shows that commute simply applies regardless of the underlying value. As a result, you may see duplicate values and gaps in your sequence (shown in the 90s in our output). If you wanted to ensure that the value didn't change during your transaction you could switch to alter. The following example shows the behavior of changing from commute to alter.user=> (def uid (ref 0))
#'user/uid
user=> (def used-id (ref []))
#'user/used-id
user=> (defn use-id []
(dosync
(alter uid inc)
(alter used-id conj @uid)))
#'user/use-id
user=> (dothreads use-id :threads 10 :times 10)
nil
user=> @used-id
[1 2 3 4 5 6 7 8 9 10 ... 91 92 93 94 95 96 97 98 99 100]There are more advanced examples using refs in The Joy of Clojure for those of you looking to discuss corner case conditions.

Last, but not least, agents are also available. From clojure.org/agents -

Like Refs, Agents provide shared access to mutable state. Where Refs support coordinated, synchronous change of multiple locations, Agents provide independent, asynchronous change of individual locations.

While I understand agents conceptually, I haven't used them much in practice. Some people love them, and the last team I was on switched to using agents heavily in one of our applications shortly after I left. But, I personally don't have enough experience to say exactly where I think they fit in. I'm sure that will be a topic for a future blog post.

Between Rich's essay and the examples above I hope a few things have become clear:Clojure has plenty of support for managing stateRich's distinction between identity and value allows Clojure to benefit from immutable structures while also allowing identity reassignment.Clojure, it's about state.
© Jay Fields - www.jayfields.com

Readable Clojure Without a Java Equivalent?

I've recently joined a new team and we've been doing a bit of Clojure. If you've never done Lisp (and I hadn't before I found Clojure) it's natural to ask the quesetion: Will programming in Lisp, especially Prefix Notation, ever feel natural? The question came up a few weeks ago and I had two answers

First of all, I've never been really upset about parenthesis. In fact, I've been doing a bit of Java these days and I don't see much difference between verify(foo).bar(eq(100), any(Baz.class), eq("Cat")) and (-> foo verify (.bar (eq 100) (any Baz) (eq "Cat))). By my count it's the same number of parenthesis. Where they're located moves around a bit, but I don't consider that to be a good or a bad thing.

People also like to bring up examples like this:(apply merge-with
(pmap count-lines
(partition-all *batch-size*
(line-seq (reader filename)))))Stefan Tilkov addressed this in a previous blog entry, and (in the comments of Stefan's entry) Rich Hickey points out that you can use a few different versions if you prefer to lay your code out in a different manner. Below are two alternative solutions Rich provided.; The same code in a pipelined Clojure style:

(->> (line-seq (reader filename))
(partition-all *batch-size*)
(pmap count-lines)
(apply merge-with ))

; and in a step-wise Clojure style with labeled interim results (a la Adrian's comment):

(let [lines (line-seq (reader filename))
processed-data (pmap count-lines
(partition-all *batch-size* lines))]
(apply merge-with processed-data))I've felt the same way for awhile. You can write Cobol in any language. The real question is, are you up for learning how to solve problems elegantly and idiomatically in a new language. If you're willing to invest the time, you'll be able to find out for yourself if it feels natural to write idiomatic Clojure code.

That was my answer, until today. While working on some Java code I stumbled on the following Java method.public void onFill(int fillQty) {
this.qty = Math.max(this.qty - fillQty, 0);
}This is a simple Java method that is decrementing the outstanding quantity state of an order by the amount of the order that just been filled. While reading the line I couldn't help but feel like there should be a more elegant way to express the logic. You want to set the outstanding quantity state to the current outstanding quantity minus what's just been filled, but you also never want the outstanding quantity to go below zero. I read from left to right, and I really wanted a way to express this logic in a way that followed the left to right pattern.

In Clojure, this is easy to do:(swap! qty #(-> % (- fill-qty) (Math/max 0)))For readers who are less familiar with Clojure dispatch reader macro, the above example can also be written as:(swap! qty (fn [current-qty] (-> current-qty (- fill-qty) (Math/max 0))))In the example above swap! is setting the qty state with the return value of the function.

If you're really new to Clojure, that might still be too much to take, so we can reduce the example and remove the state setting. Here's the version in Java that ignores setting state.Math.max(this.qty - fillQty, 0);The example below is a logical equivalent in Clojure.(-> qty (- fill-qty) (Math/max 0))When reading the above Java example I'm forced to put Math.max on my mental stack, evaluate this.qty - fillQty, and then mentally evaluate the method I put on the stack with my new result and the additional args. This isn't rocket science, but it's also not how I naturally read (left to right). On the other hand, when I read the Clojure version I think - take the current quantity, subtract the fill quantity, then take the max of that and zero. The code reads in small, logical chucks that are easy for me to digest.

Obviously, the Java code can also be rewritten in a few other ways. Here's an example of Java code that reads left to right and top to bottom.public void onFill(int fillQty) {
this.qty -= fillQty
this.qty = Math.max(this.qty, 0);
}And, I can do something similar in Clojure, if I want.(swap! qty #(- % fill-qty))
(swap! qty #(Math/max % 0))While it's possible to write Clojure similar to the above example, it's much more likely that you'd use a let statement if you wanted to break up the two operations.(defn update-qty [current fill-qty]
(let [total-qty (- current fill-qty)]
(Math/max total-qty 0)))

(swap! qty update-qty fill-qty)The above example is probably about equivalent to the following Java snippet.public void onFill(int fillQty) {
int newTotalQty = this.qty - fillQty
this.qty = Math.max(newTotalQty, 0);
}So, I can write code that is similar to my options in Java, but I'm still left wanting a Java version that is similar to this Clojure example:(swap! qty #(-> % (- fill-qty) (Math/max 0)))The only thing that springs to mind is some type of fluent interface that allows me to say this.qty = this.qty.minus(fillQty).maxOfIdentityOrZero(), but I can't think of a realistic way to create that API without quite a bit of infrastructure code (including my own Integer class).

(note, you could extend Integer in a language with open classes, but that's outside the scope of this discussion)

The last Clojure example is definitely the version of the code I would prefer. My preference is based on the way the code reads in concise, logical chunks from left to write. I don't have to solve inside out like the original java version forces me to, and I don't have to split my work across two lines.

I'm sure there are situations where Java allowed me to create an elegant solution that wouldn't have been possible in Clojure. This entry isn't designed to send a "Clojure is better than Java" message. I don't believe that. However, before today I've held the opinion that you can write Clojure that logically breaks up problems in ways very similar to what you do using Java. However, I've now also expanded my opinion to include the fact that in certain situations Clojure can also allow me to solve problems in a way that I find superior to my options within Java.

And, yes, after a bit of time getting used to Lisp syntax, it definitely does feel perfectly natural to me when I'm developing using Clojure.
© Jay Fields - www.jayfields.com

Types of Technical Debt

As a developer at DRW, technical debt is often on your mind. Our front office teams work directly with their traders (often sitting directly next to them), and are exposed in real-time to their software needs. While sitting with the traders you see the immediate impact of your recent feature additions. The feedback loop is extremely tight - In the past I've come in to work, received a feature request early in the trading session, coded the change, deployed it to production, and seen the profits as part of that same trading day. In that kind of environment, you tend to care more about adding features than you do about having the most elegantly designed software in the history of the world.

But, you can't ignore code quality due to it's impact on delivery speed. There's a constant internal personal struggle between delivering features and keeping quality at a level that enables high speed delivery. Our technical debt is very visible, we keep reminders around to ensure we address our technical debt in the future. We also use those same reminders to prioritize which technical debt payoff will most benefit the team. The traders are also very aware of the technical debt, and it's often the case that they ask the question: Do you guys need to spend some time paying down this technical debt?

Working in an environment where the traders understand technical debt and embrace it's payoff is great, but it exposes other issues with technical debt.

Steve Freeman's blog entry, Bad code isn't Technical Debt, it's an unhedged Call Option, first got me thinking about different types of technical debt. That blog entry rang very true for me. I really liked the idea that selling a call option was a better metaphor; however, I wondered if it was something that could easily work in your average workplace.

I happen to work in a trading firm, so my customers know the implications of selling a call option. But, the entire idea of a metaphor is to understand one thing in terms of another. If you don't know either things, the metaphor doesn't really work. I don't think the average customer already understands options; therefore, I don't think the new metaphor will ever catch on.

But, I did believe it was a superior metaphor. So, I started looking at my bad code as selling calls instead of debt. In some cases the metaphor worked perfectly. We created an analysis tool that was mildly important to the traders and didn't have strict reliability needs. This analysis tool didn't have a single test. If the tool changed it's role, we would have had to put in a fair bit of work to ensure that it was more bug resilient. However, the tool served it's purpose and at this point I don't believe it's used by anyone. We saved a bunch of time (collected the premium) without ever having to make significant changes to the code (deliver).

However, in other cases the 'selling calls' metaphor didn't fit as well. In general, developers tend to call anything that slows them down 'techincal debt'; therefore, a slow running test suite would be considered technical debt. If everything that used to be called technical debt is now selling a call, then a slow running test suite is also selling a call. Except, that doesn't fit. I pay the time it takes to run the test suite several times a day. In the case of a slow running test suite, I think a high interest loan is a better metaphor.

Therefore, if all things that slow a developer down are 'technical debt' then you have some debt where interest is paid on an ongoing basis (slow running tests) and other debts where you'll only pay under specific circumstances (extending an application with no tests). I think this is a concept that is understood intuitively by most developers, but I wonder if adding visual type indicators to the identified technical debt would help with understanding and prioritization.

As a customer, I can imagine I'd appreciate knowing what is costing me on a daily basis and what isn't costing me anything today, but could cripple me tomorrow. And, as a team member I would be interested in highlighting ongoing pain and moving that higher on the priority list (assuming the other risks were reasonable). I imagine team members are already doing this subconsciously, but making it explicit could provide valuable to other members of the development team that have less context on specific situations.

In thinking about different types of technical debt, I wondered if there were other types of technical debt I was overlooking or incorrectly grouping with "high interest loans" or "selling calls". If any ideas spring to mind, please leave a comment.
© Jay Fields - www.jayfields.com

Random Thoughts on Good Programmers and Good Code

I've know some really, really smart guys who write code I hate. I love using their libraries, but I don't ever want to edit their source. To me, their source is too complicated. I used to think that they were smarter than I was and I benefited by having to write clearer code, as I couldn't easily digest their complicated code.

But, I've also run into code I've written that is complicated in it's own ways. I like to find the dark corners of languages and exploit them. To me, 8 different classes being configured 10 different ways and used in 3 different situations is something I can never get comfortable with. However, metaprogramming & macros don't scare me at all, and I often (over) embrace them.

And, to me, that's the heart of the issue. Each of us have a comfort level with different styles of development. I like to write as little as possible, and embrace "magic" that allows me to write less. Other people are more comfortable with more code, as long as it follows patterns that are easy to follow through the system. We could decide which is better for "average" programmers, but I don't think there's a "better" approach for good programmers who have tried both and stick with what they prefer; they know to play to their strengths.

This also helps explain why all code we haven't written "sucks" and all code we write is "awesome"

As an example, I think test names are worthless because I don't use them. Dan North loves test names and uses them to help him understand the responsibility of code. It's as simple as a different approach for digesting code, but the result is assigning a value to an artifact. I can write all I want about how test names are worthless, but it's not going to change the fact that they are helpful to Dan. He and I could (ignorantly) argue till the end of time about the value of test names, but the actual discussion is in how we digest code.

These days, the Java, Ruby, Clojure, Patterns, Metaprogramming, Static, Dynamic, etc discussions all feel this way to me. You're either an idiot who doesn't understand what you're arguing against, or you're talented enough that you found what makes you the most productive. Either way, your opinions don't necessarily apply to anyone else.

That said, I think there is value in presenting new ideas (or restating old ideas in new ways) as it can inspire people to reevaluate their assumptions. The ideas that tend to be most helpful for me are the ones presented as experience reports and discussions on the benefits of a particular solution. The ideas that are presented as an X vs Y smackdown always tend to make me wonder if someone is still searching for a silver bullet.
© Jay Fields - www.jayfields.com

Clojure: Eval-ing a String in Clojure

I recently needed to eval a string into Clojure and found it was easy, but it wasn't accomplished in the way I expected. My recent experience with eval was in Ruby, so I naturally reached for Clojure's eval function. I quickly found that Clojure's eval was not what I was looking for, well, not exactly what I was looking for.(eval "(+ 1 1)")
"(+ 1 1)"This quick trip to the REPL reminded me of the last time I needed to eval a string, and the previous experiences I'd had with read-string and load-string. It turns out load-string does the trick, and read-string + eval can also get the job done.user=> (load-string "(+ 1 1)")
2
user=> (eval (read-string "(+ 1 1)"))
2Looking at the documentation for read-string and load-string can help point you at which you might need for your given situation.

read-string: Reads one object from the string [argument]
load-string: Sequentially read[s] and evaluate[s] the set of forms contained in the string [argument]

Okay, it looks like you'd probably want load-string if you had multiple forms. Back to the REPL.user=> (load-string "(println 1) (println 2)")
1
2
user=> (eval (read-string "(println 1) (println 2)"))
1As you can see, load-string evaluated the entire string and read-string only returned the first form to eval. The last examples show printing values (and I removed the nil return values), but it's more likely that you'll be concerned with returning something from read-string or load-string. The following example shows what's returned by both functions.user=> (load-string "1 2")
2
user=> (eval (read-string "1 2"))
1Given our previous results, I'd say load-string and read-string are returning the values you'd expect. It seems like it's easy to default to load-string, but that's not necessarily the case if evaluation time is important to you. Here's a quick snippet to get the point across.user=> (time (dotimes [_ 100] (eval (read-string "1 2"))))
"Elapsed time: 12.277 msecs"
nil
user=> (time (dotimes [_ 100] (load-string "1 2")))
"Elapsed time: 33.024 msecs"
nilOne of the applications I previously worked on would write a large number of events to a log each day; the format used to log an event was a Clojure map. If the application was restarted the log would be read and each event would be replayed to get the internal state back to current. We originally started by using load-string; however, start-up time was becoming a problem, and a switch to read-string completely removed the issue.

Like so many examples in programming, there's a reason that two versions exist. It's worth taking the time to see which solution best fits your problem.
© Jay Fields - www.jayfields.com

Clojure: if-let and when-let

I'm a fan of if-let and when-let. Both can be helpful in creating succinct, readable Clojure code. From the documentation:

If test is true, evaluates then with binding-form bound to the value of test, if not, yields else

A quick example should demonstrate the behavior.user=> (if-let [a :anything] a "no")
:anything
user=> (if-let [a nil] a "no")
"no"
user=> (if-let [a false] a "no")
"no"The above example demonstrates a simple case where a truthy value causes the "then" form to be returned, and a falsey value (nil or false) causes the "else" form to be returned. The example also shows that "a" is bound to the value :anything and can be used within the "then".

The following example shows that you can put code in the "then" and "else" statement's as well.user=> (if-let [a 4] (+ a 4) (+ 10 10))
8
user=> (if-let [a nil] (+ a 4) (+ 10 10))
20Just like above, when "a" is truthy the value is bound and it can be used as desired in the "then".

The when-let function behaves basically the same way, except there is no "else".user=> (when-let [a 9] (+ a 4))
13
user=> (when-let [a nil] (+ a 4))
nil
user=> (when-let [a 9] (println a) (+ a 4))
9
13The example demonstrates that then "then" case is very similar to if-let; however, if the test fails the when-let function simply returns nil. The last when-let example also shows how it slightly differs from an if-let: you can pass as many forms to the "then" as you'd like. If the test evaluates to true, all of the additional forms will be evaluated.

There are a few things worth knowing about if-let and when-let with respect to the bindings. Both if-let and when-let require a vector for their bindings, and there must be exactly two forms in the binding vector. The following example shows Clojure's response if you choose not to follow those rules.user=> (when-let (a 9) (println a))
java.lang.IllegalArgumentException: when-let requires a vector for its binding (NO_SOURCE_FILE:0)
user=> (when-let nil (println "hi"))
java.lang.IllegalArgumentException: when-let requires a vector for its binding (NO_SOURCE_FILE:0)
user=> (when-let [a 9 b nil] (println a b))
java.lang.IllegalArgumentException: when-let requires exactly 2 forms in binding vector (NO_SOURCE_FILE:0)It's nice to know if-let and when-let, but they aren't exactly hard concepts to follow. Once someone points them out to you, I expect you'll be able to easily use them in your code without much effort.

However, what is the impact of destructuring on if-let and when-let? I know what I expected, but I thought it was worth a quick trip to the REPL to make sure my assumptions were correct. The following code shows that as long as the value being bound is truthy the "then" will be executed - destructuring values have their normal behavior and do no effect execution flow.user=> (when-let [{a :a} {:a 1}] [a])
[1]
user=> (when-let [{a :a} {}] [a])
[nil]
user=> (when-let [{a :a} {:a false}] [a])
[false]
user=> (when-let [{a :a} nil] [a])
nilAs you would expect the when-let is consistent and the destructuring behavior is also consistent.

if-let and when-let are good, give them a shot.
© Jay Fields - www.jayfields.com

Clojure: Truthy and Falsey

Truthy and Falsey are important concepts in Clojure. The best description I've seen is in The Joy of Clojure, go buy it.

Depending on your language background you may have very different ideas on what evaluates to true and what evaluates to false. In Clojure, the answer to this is very straightforward.

Both nil and false are treated as "false" and everything else is treated as true.

Let's hit the REPL.user=> (if true "yes" "no")
"yes"
user=> (if false "yes" "no")
"no"Okay, that much should be obvious. However, it's important to note that nil is also considered false, which is why we say:

In Clojure nil and false are considered "false" and therefore we say they are both "falsey".

Another quick trip to the REPL to verify that nil is falsey.user=> (if nil "yes" "no")
"no"As expected, the else is evaluated.

There are other values that are considered false in other languages, this does not apply to clojure.

All non-falsey values are considered "truthy" and evaluate as such.

Here are some common examples of truthy values in Clojure that are falsey in other languages.user=> (if -1 "still true" "false")
"still true"
user=> (if 0 "still true" "false")
"still true"
user=> (if [] "still true" "false")
"still true"
user=> (if (list) "still true" "false")
"still true"As it says in The Joy of Clojure: Every[thing] is "true" all the time, unless it is nil or false.
© Jay Fields - www.jayfields.com

The Impact of Accidental Complexity on Estimates

At speakerconf Aruba, J.B. Rainsberger recently gave an enlightening talk which included discussion on the impact of accidental complexity to estimates.

In his talk, he pointed out that estimates are composed of accidental and essential complexity. It's often easy to estimate the essential complexity of a problem; however, estimating accidental complexity is an entirely different matter. This point really hit home for me. On my last project I often strongly disagreed with certain members of the team on estimates. I was often higher than the average, and much higher than the lowest estimates. J.B.'s talk really openend my eyes to a few points.Not all developers consider both accidental and essential complexity while estimatingFeatures that touch accidentally complex portions of the system are harder to estimate as the accidental complexity can often be hard to completely understand. (If you completely understood it, you'd probably remove the accidental complexity)Introducing technical debt increases accidental complexity, and as a side-effect invalidates previous estimates and increases the likelihood that future estimates will differ in size drastically.The first two points had always been intuitive to me; however, it was good to hear the ideas clearly articulated from someone else's point of view. The last point was the one that really interested me.

I'm lucky enough to work for a company where I don't need to explain the benefits of addressing technical debt. As developers we sometimes consciously introduce technical debt when it will allow us a short-term benefit, and we address that technical debt when appropriate. I've always considered technical debt to be part of the developer's world, and I only mention it to the business when completing a feature takes longer than expected due to tech debt. I never considered the impact of technical debt on my ability to produce accurate estimates for the business.

J.B.'s talk made me ask myself a few questions about decisions I'd been making. Most importantly: If the business knew that I was taking a short-cut to deliver sooner, but it was going to damage estimates, would that be acceptable? I think that's an important question to consider if your client values estimates, and it's something I'll continue to ask myself in the future.
© Jay Fields - www.jayfields.com

The President and the CEO of an Application

The CEO is a visionary, often leaving day-to-day operations to the President -- wikipedia

I recently switched teams. As part of the transition process, I started taking on less and less of the responsibility for things that were long-term goals. I was still very interested in helping the team succeed, but not at the risk of leaving them with something that would be harder to maintain.

So, I found myself sitting with the users and looking for the lowest-risk, highest-impact features that I could work on for my final days. This seemed like the responsible thing to do for the long-term health of the project, but an interesting side-effect was the unexpected impact to the short-term success of the project.

At first I started grabbing the story-cards that fit this criteria, and I worked my way from the highest prioritized on down. As you can imagine, I moved down the list quickly - many of the cards were sufficiently complicated that working on them would have created maintenance risk. The users were pleased with the quick additions of highly prioritized features; however, as I moved down the list their happiness became very erratic. Some features were still very appreciated; while others were met largely with ambivalence. It became obvious that I needed to adapt the plan.

I stopped working in priority order. In fact, I stopped working from the story cards entirely. I sat down next to the users twice a day, for extended periods of time, listened to anything they had to say, and I watched the way the application was used. I started making tweaks to the application based on their complaints and any wasted effort that I witnessed. I looked for "every time I do A, then I do B". Sometimes the interaction was complicated and I couldn't automate it away. However, there were non-trivial amounts of tasks that I entirely eliminated.

The users were great about providing feedback. There was some feedback that I had to dismiss, because it wasn't something I could fix safely (in the short-term); however, a large amount of their complaints took 5-20 minutes to address. Those issues were nagging and annoying, but not (individually) more important than the long term goals of the project. But, as I began to quickly address the issues the application became more polished. It went from a good application, to an application that was surprisingly nice to work with. One of the most demanding users I've ever worked for sent me a few different emails letting me know how great the app was and how much he appreciated my effort. I was shocked.

Delivering all of these little, helpful features so greatly impacted the happiness and effectiveness of the users that I had to take a step back and see if this was something I could reproduce on future projects. The first thing that came to mind was the relationship between a CEO and a President. As the quote above says, The CEO is a visionary, often leaving day-to-day operations to the President. This idea seemed similar to what occurred in my final days. I was adding features that made day-to-day activities more efficient, and at the same time a pair was working on the features that solved our long term goals. The application was moving in the right direction for the long-term, while still experiencing short bursts of improvement in the short-term as well.

It wasn't unfiltered awesomeness. I delivered a few things that addressed an immediate problem, but not in a way that satisfied the users' needs. I wasn't working off story cards or deep conversations. I was often speculating on what I thought the application needed most. Sometimes I was wrong. However, each failure occurred quickly, and led to delivering something that the users did need. And, this cycle was complete within a few hours. (e.g. [in an hour] deliver a feature that isn't quite right, get immediate feedback, spend an hour fixing it, deliver what's really needed)

You could argue that I wasted that first hour, but spending 30 minutes with 2 users to get it right the first time would have cost the team more in the end. But, we're quickly approaching dangerous territory. Developers may like the idea of no requirements and delivering speculative features all-day, but I expect businesses would (rightly-so) object. I think a key variable in my scenario was the 2 hour failure-success cycle. If we tried this approach, it took 2 weeks, and it could have been avoided by a 30 minute discussion I think everyone would consider it a failure (again, rightly-so).

This led me to the observation that developers are more likely to correctly prioritize features that take less than a day to deliver and users are more likely to correctly prioritize features based on long-term goals. I've definitely seen this occur in two ways in the past. I often see users prioritize based solely on the value of the feature, even when an estimate is present.I have personally thought prioritizing some small features over a larger feature was the right choice, only to see that the impact of the large feature was significantly greater than I understoodThis is also intuitive to me; developers are more likely to correctly estimate a smaller change and users are more likely to correctly estimate the impact of a larger change.

This experience has definitely left me wondering if teams would benefit from:developers prioritizing and keeping a separate backlog of smaller feature requestsusers prioritizing and keeping a separate backlog of larger feature requestsensuring there is always at least one person working on each backlog.I was very pleased with the results of using this style, and I'm hoping to try it again in the future.
© Jay Fields - www.jayfields.com