Daniel Higginbotham's Blog

Frameworks and Why (Clojure) Programmers Need Them

It seems like there's a strong aversion to using frameworks in the
Clojure community. Other languages might need frameworks, but not
ours! Libraries all the way, baby!

This attitude did not develop without reason. Many of us came to
Clojure after getting burned on magical frameworks like Rails, where
we ended up spending an inordinate amount of time coming up with hacks
for the framework's shortcomings. Another "problem" is that Clojure
tools like Luminus and the top-rate web
dev libraries it bundles provide such a productive experience that
frameworks seem superfluous.

Be that as it may, I'm going to make the case for why the community's
dominant view of frameworks needs revision. Frameworks are useful. To
convince you, I'll start by explaining what a framework is. I have yet
to read a definition of framework that satisfies me, and I think
some of the hate directed at them stems from a lack of clarity about
what exactly they are. Are they just glorified libraries? Do they have
to be magical? Is there some law decreeing that they have to be more
trouble than they're worth? All this and more shall be revealed.

I think the utility of frameworks will become evident by describing
the purpose they serve and how they achieve that purpose. The
description will also clarify what makes a good framework and
explain why some frameworks end up hurting us. My hope is that you'll
find this discussion interesting and satisfying, and that it will give
you a new, useful perspective not just on frameworks but on
programming in general. Even if you still don't want to use a
framework after you finish reading, I hope you'll have a better
understanding of the problems frameworks are meant to solve and that
this will help you design applications better.

Frameworks have second-order benefits, and I'll cover those too. They
make it possible for an ecosystem of reusable components to
exist. They make programming fun. They make it easier for beginners to
make stuff.

Last, I'll cover some ways that I think Clojure is uniquely suited to
creating kick-ass frameworks.

(By the way: I've written this post because I'm building a Clojure
framework! So yeah this is totally my Jedi mind trick to prime you to
use my framework. The framework's not released yet, but I've used it
to build Grateful Place, a community for people who are into
cultivating gratitude, compassion, generosity, and other positive
practices. Just as learning Clojure makes
you a better programmer, learning to approach each day with
compassion, curiosity, kindness, and gratitude will make you a more
joyful person. If you want to brighten your day and mine, please
join!)

What is a Framework?

A framework is a set of libraries that:

Manages the complexity of coordinating the resources needed to
write an application...
by providing abstractions for those resources...
and systems for communicating between those resources...
within an environment...
so that programmers can focus on writing the business logic that's
specific to their product

I'll elaborate on each of these points using examples from
Rails and from the ultimate framework: the
operating system.

You might wonder, how is an OS a framework? When you look at the list
of framework responsibilities, you'll notice that the OS handles all
of them, and it handles them exceedingly well. Briefly: an OS provides
virtual abstractions for hardware resources so that programmers don't
have to focus on the details of, say, pushing bytes onto some
particular disk or managing CPU scheduling. It also provides the
conventions of a hierarchical filesystem with an addressing system
consisting of names separated by forward slashes, and these
conventions provide one way for resources to communicate with each
other (Process A can write to /foo/bar while Process B reads from
it) - if every programmer came up with her own bespoke addressing
system, it would be a disaster. The OS handles this for us so we can
focus on application-specific tasks.

Because operating systems are such successful frameworks we'll look at
a few of their features in some detail so that we can get a better
understanding of what good framework design looks like.

Coordinating Resources

Resources are the "materials" used by programs to do their work, and
can be divided into four categories: storage, computation,
communication, and interfaces. Examples of storage include files,
databases, and caches. Computation examples include processes,
threads, actors, background jobs, and core.async processes. For
communication there are HTTP requests, message queues, and event
buses. Interfaces typically include keyboard and mouse, plus screens
and the systems used to display stuff on them: gui toolkits, browsers
and the DOM, etc.

Specialized resources are built on top of more general-purpose
resources. (Some refer to these specialized resources as services or
components.) We start with hardware and build virtual resources on
top. With storage, the OS starts with disks and memory and creates the
filesystem as a virtual storage resource on top. Databases like
Postgres use the filesystem to create another virtual storage resource
to handle use cases not met by the filesystem. Datomic uses other
databases like Cassandra or DynamoDB as its storage layer. Browsers
create their own virtual environments and introduce new resources like
local storage and cookies.

For computation, the OS introduces processes and threads as virtual
resources representing and organizing program execution. Erlang
creates an environment with a process model that's dramatically
different from the underlying OS's. Same deal with Clojure's
core.async, which introduces the communicating sequential
processes computation model. It's a virtual model defined by Clojure
macros, "compiled" to core clojure, then compiled to JVM bytecode (or
JavaScript!), which then has to be executed by operating system
processes.

Interfaces follow the same pattern: on the visual display side, the OS
paints to monitors, applications paint to their own virtual canvas,
browsers are applications which introduce their own resources (the DOM
and <canvas>), and React introduces a virtual DOM. Emacs is an
operating system on top of the operating system, and it provides
windows and frames.

Resources manage their own entities: in a database, entities could
include tables, rows, triggers, and sequences. Filesystem entities
include directories and files. A GUI manages windows, menu bars, and
other components.

(I realize that this description of resource is not the kind of
airtight, axiomatic, comprehensive description that programmers like.
One shortcoming is that the boundary between resource and application
is pretty thin: Postgres is an application in its own right, but from
the perspective of a Rails app it's a resource. Still, hopefully my
use of resource is clear enough that you nevertheless understand
what the f I'm talking about when I talk about resources.)

Coordinating these resources is inherently complex. Hell, coordinating
anything is complex. I still remember the first time I got smacked in
the face with a baseball in little league thanks to a lack of
coordination. There was also a time period where I, as a child, took
tae kwon do classes and frequently ended up sitting with my back
against the wall with my eyes closed in pain because a) my mom for
some reason refused to buy me an athletic cup and b) I did not possess
the coordination to otherwise protect myself during sparring.

When building a product, you have to decide how to create, validate,
secure, and dispose of resource entities; how to convey entities from
one resource to another; and how to deal with issues like timing (race
conditions) and failure handling that arise whenever resources
interact, all without getting hit in the face. Rails, for instance,
was designed to coordinate browsers, HTTP servers, and databases. It
had to convey user input to a database, and also retrieve and render
database records for display by the user interface, via HTTP requests
and responses.

There is no obvious or objectively correct way to coordinate these
resources. In Rails, HTTP requests would get dispatched to a
Controller, which was responsible for interacting with a database and
making data available to a View, which would render HTML that could be
sent back to the browser.

You don't have to coordinate web app resources using the
Model/View/Controller (MVC) approach Rails uses, but you do have to
coordinate these resources somehow. These decisions involve making
tradeoffs and imposing constraints to achieve a balance of
extensibility (creating a system generic enough for new resources to
participate) and power (allowing the system to fully exploit the
unique features of a specific resource).

This is a very difficult task even for experienced developers, and the
choices you make could have negative repercussions that aren't
apparent until you're heavily invested in them. With Rails, for
instance, ActiveRecord (AR) provided a good generic abstraction for
databases, but early on it was very easy to produce extremely
inefficient SQL, and sometimes very difficult to produce efficient
SQL. You'd often have to hand-write SQL, eliminating some of the
benefits of using AR in the first place.

For complete beginners, the task of making these tradeoffs is
impossible because doing so requires experience. Beginners won't even
know that it's necessary to make these decisions. At the same time,
more experienced developers would prefer to spend their time and
energy solving more important problems.

Frameworks make these decisions for us, allowing us to focus on
business logic, and they do so by introducing communication systems
and abstractions.

Resource Abstractions

Our software interacts with resources via their abstractions. I
think of abstractions as:

the data structures used to represent a resource
the set of messages that a resource responds to
the mechanisms the resource uses to call your application's code

(Abstraction might be a terrible word to use here. Every developer
over three years old has their own definition, and if mine doesn't
correspond to yours just cut me a little slack and run with it :)

Rails exposes a database resource that your application code interacts
with via the ActiveRecord abstraction. Tables correspond to classes,
and rows to objects of that class. This a choice with tradeoffs - rows
could have been represented as Ruby hashes (a primitive akin to a JSON
object), which might have made them more portable while making it more
difficult to concisely express database operations like save and
destroy. The abstraction also responds to find, create,
update, and destroy. It calls your application's code via
lifecycle callback methods like before_validation. Frameworks add
value by identifying these lifecycles and providing interfaces for
them when they're absent from the underlying resource.

You already know this, but it bears saying: abstractions let us code
at a higher level. Framework abstractions handle the concerns that are
specific to resource management, letting us focus on building
products. Designed well, they enable loose coupling.

Nothing exemplifies this better than the massively successful file
abstraction that the UNIX framework introduced. We're going to look at
in detail because it embodies design wisdom that can help us
understand what makes a good framework.

The core file functions are open, read, write, and
close. Files are represented as sequential streams of bytes, which
is just as much a choice as ActiveRecord's choice to use Ruby
objects. Within processes, open files are represented as file
descriptors, which are usually a small integer. The open function
takes a path and returns a file descriptor, and read, write, and
close take a file descriptor as an argument to do their work.

Now here's the amazing magical kicker: file doesn't have to mean
file on disk. Just as Rails implements the ActiveRecord abstraction
for MySQL and Postgres, the OS implements the file abstraction for
pipes, terminals, and other resources, meaning that your programs
can write to them using the same system calls as you'd use to write
files to disk - indeed, from your program's standpoint, all it knows
is that it's writing to a file; it doesn't know that the "file" that a
file descriptor refers to might actually be a pipe.

Exercise for the reader: write a couple paragraphs explaining
precisely the design choices that enable this degree of loose
coupling. How can these choices help us in evaluating and designing
frameworks?

This design is a huge part of UNIX's famed simplicity. It's what lets
us run this in a shell:

# list files in the current directory and perform a word count on the output
ls | wc

The shell interprets this by launching an ls process. Normally, when
a process is launched it creates three file descriptors (which,
remember, represent open files): 0 for STDIN, 1 for STDOUT,
and 2 for STDERR, and the shell sets each file descriptor to refer
to your terminal (terminals can be files!! what!?!?). Your shell sees
the pipe, |, and sets ls's STDOUT to the pipe's STDIN, and the
pipe's STDOUT to wc's STDIN. The pipe links processes' file
descriptors, while the processes get to read and write "files" without
having to know what's actually on the other end. No joke, every time I
think of this I get a little excited tingle at the base of my
spine because I am a:

This is why file I/O is referred to as the universal I/O model. I'll
have more to say about this in the next section, but I share it here
to illustrate how much more powerful your programming environment can
be if you find the right abstractions. The file I/O model still
dominates decades after its introduction, making our lives easier
without our even having to understand how it actually works.

The canonical first exercise any beginner programmer performs is to
write a program that prints out, Wassup, homies?. This program makes
use of the file model, but the beginner doesn't have to even know that
such a thing exists. This is what a good framework does. A
well-designed framework lets you easily get started building simple
applications, without preventing you building more complicated and
useful ones as you learn more.

One final point about abstractions: they provide mechanisms for
calling your application's code. We saw this a bit earlier with
ActiveRecord's lifecycle methods. Frameworks will usually provide the
overall structure for how an application should interact with its
environment, defining sets of events that you write custom handlers
for. With ActiveRecord lifecycles, the structure of before_create,
create, after_create is predetermined, but you can define what
happens at each step. This pattern is called inversion of control,
and many developers consider it a key feature of frameworks.

With *nix operating systems, you could say that in C programs the
main function is a kind of onStart callback. The OS calls main,
and main tells the OS what instructions should be run. However, the
OS controls when instructions are actually executed because the OS is
in charge of scheduling. It's a kind of inversion of control, right?

Buddy Systems at Work: a Framework for Rewarding Relationships on Remote Teams

One of my work mottos is "support the hell out of each other": helping
your friends grow and succeed is far more rewarding than chasing after
the next gold star for yourself. While I was at Reify
Health I was lucky enough to work with
other devs who had a similar ethos. There were no rock stars, and
everyone looked out for each other. I'm happy to have gotten to work
somewhere where I made real, honest-to-god friends :)

During my final weeks there I collaborated with coworkers on an
experimental framework for supporting each other in a formal
context. This framework would let us explore how to be there for each
other in more meaningful ways while maintaining boundaries. This
seemed particularly valuable in Reify's mostly-remote
environment. With their permission, I'm sharing the initial framework
proposal below:

The Buddy System

This doc outlines a proposal for a buddy system. The main idea behind the buddy system is for us to support each other by regularly spending time chatting with each other about things beyond whatever story is at hand. Exactly what that looks like will be up to you; this doc includes suggestions for how to have rewarding conversations. First, here are some ground rules:

Ground Rules


It’s confidential. Whatever you talk about remains between you and your buddy unless you say otherwise. This is meant to provide each other with time and space to speak about what’s going on :D
It’s completely voluntary. You don’t have to participate, and if you do participate you can shape it however you want to: the frequency and content of chats is up to you.
You can opt out or change buddies whenever you want. If you decide to drop out of the buddy system, it’s totally up to you if you want to explain why, or not. The buddy system is meant to consciously cultivate a space where we can be vulnerable with each other; part of making that safe is setting the expectation that people can opt out without having to explain why.
By the same token, in joining the buddy system you accept that your buddy may at some point wish to opt or change buddies, and you commit to try to not take that personally if it happens

Techniques for Efficiently Learning Programming Languages

Learning programming languages is a skill: do it well and you'll
experience one dopamine hit after another as you master something new;
do it poorly and you'll feel constantly frustrated and even give
up. What follows are the best techniques for learning programming
languages that I've picked up over years of teaching programming by
writing books and
articles,
doing talks, and
running
a training course. Many of
these techniques are pulled from books explaining the latest research
in efficient learning, and you can find those books (along with other
great programming books) at
Community Picks: Learn Programming.

Test Yourself Constantly to Defeat The Illusion of Competence

One of the worst ways to learn is to re-read or re-watch
material. This kind of review gives you the feeling that you
understand the topic covered because it seems like you're
understanding the topic effortlessly. Researchers call this the
illusion of competence.

A significantly better approach (and one of the best techniques you
can employ) is to test yourself constantly. Instead of re-reading
what a function or class or object is, ask yourself to define these
concepts or use them in a short program; force yourself to somehow
demonstrate your understanding. This process often feels
uncomfortable, but it's much more efficient at forming long term
memories. You can take this one step further and test yourself before
you've covered the material by, for example, attempting exercises
before reading a chapter. Remarkably, this has also been shown aid
memory formation.

The impressive impact that testing has on learning is called the
testing effect, and here are some specific ways you can take
advantage of it:

Before reading a chapter or watching a video, try guessing at what
you're about to learn and write it down.
Try doing a chapter's exercises before reading the chapter.
Always do exercises, even the hard ones. It's OK to give up on an
exercise and come back to it later (or never, even), but at least
try it. (More on this in the next section.)
Read a short program and try to recreate it without looking at the
original code. Or, go smaller and do this with a function.
Immediately after learning a new concept like objects, classes,
methods, or higher-order functions, write code that demonstrates
that concept.
Create diagrams that illustrate concepts, both in isolation and how
they relate to each other.
Blog about a concept you just learned.
Try explaining the concept to a non-technical friend. (I did this a
lot when writing Clojure for the Brave and True; being able to
explain an idea in layman's terms forces you to understand the idea
deeply.)

Many of these techniques boil down to write some code! With
programming it's easy to believe we're learning a lot just by reading
because programming is text-heavy and conceptual. But it's also a
skill, and like any other skill you have to perform it to get
better. Writing code is the best way to reveal your incorrect
assumptions about programming. The faster you do that, the faster you
can make corrections and improve.

If you'd like to learn more about the testing effect, check out
make it stick: The Science of Successful Learning.

Take Time to Unfocus

If you're stuck on a problem or don't understand something you just
read, try taking a walk or even a shower -- anything to enter a
relaxed, unfocused state of mind. It probably seems counterintuitive
that one of the best ways to get unstuck is to stop trying for a
little while, but it's true.

The problem is that it's easy for us to put on mental blinders when
we're focusing hard on a problem. I mean, that's pretty much what
"focus" means. But by focusing hard, we're exploring only a small
portion of the solution space. By unfocusing, our unconscious mind is
able to explore and make connections across vast swaths of our
experience.

To me it's like when you're trying to find a destination on a paper
map (remember those?). You can unconsciously become convinced that the
city you're trying to reach should be right here! in the upper-left
qudrant of the map, so you look at it over and over without
success. Then you put the map down and take a deep breath and stare at
nothing for a minute, and when you look at the map again the actual
location jumps out at you immediately.

We've all had the experience of having a sudden insight in the shower;
now you have a slightly better understanding of why that happens, and
you can employ the technique on purpose. Personally, I will actually
take a shower if I'm stuck on something, and it's remarkable how well
the technique works. And how clean I am.

If you'd like to learn more about the focused and diffuse modes of
thinking, check out
A Mind for Numbers: How to Excel at Math and Science (Even If You FLunked Algebra).

Don't Waste Time Being Frustrated

Related to the last section: don't waste time being frustrated with
code. Frustration leads us into doing stupid things like re-compiling
a program or refreshing the browser with the hope that this time it
will be magically different.

Use frustration as a signal that there's a gap in your knowledge. Once
you realize you're frustrated, it can help to take a step back and
clearly identify the problem. If you've written some code that's not
working, explicitly explain to yourself or someone else the result
that you expected. Use the scientific method and develop a hypothesis
for what's causing the unexpected behavior. Then test your
hypothesis. Try again, and if a solution still eludes you, put the
problem aside and come back to it later.

I can't tell you how many times I've thrown my laptop in disgust over
a seemingly unsolvable problem, only to look at it the next day and
have an obvious solution pop into my head immediately. This happened
last week, even.

Identify Which Programming Language Aspect You're Dealing With

Personally, I find it useful to keep in mind that when you're learning
a programming language, you're actually learning four things:

How to write code: syntax, semantics, and resource management
The language's paradigm: object-oriented, functional, logic, etc.
The artifact ecosystem: how to build and run executables and how
to use libraries

Tooling: editors, compilers, debuggers, linters

It's easy to get these four facets mixed up, with the unfortunate
result that when you run into a problem you end up looking in
completely the wrong place.

Someone who's completely new to programming, for example, might start
out by trying to build iOS apps. They might try to get their app
running on a friend's phone, only to see some message about needing a
developer certificate or whatever. This is part of the artifact
ecosystem, but an inexperienced person might see this as a problem
with how to write code. They might look at every line they wrote to
figure out the problem, when the problem isn't with their code at all.

I find it easier to learn a language if I tackle each of these aspects
systematically, and in another blog post I'll present a general list
of questions that need answering that should help you in learning any
language.

Identify the Purpose, External Model, and Internal Model

Whenever you’re learning to use a new tool, its useful to identify its
purpose, external model and internal model.

When you understand a tool's purpose, your brain gets loaded with
helpful contextual details that make it easier for you to assimilate
new knowledge. It's like working on a puzzle: when you're able to look
at a picture of the completed puzzle, it's a lot easier to fit the
pieces together. This applies to languages themselves, and language
libraries.

A tool's external model is the interface it presents and the way it
wants you to think about problem solving. Clojure’s external model is
a Lisp that wants you to think about programming as mostly
data-centric, immutable transformations. Ansible wants you to think of
server provisioning in terms of defining the end state, rather than
defining the steps you should take to get to that state.

A tool's internal model is how it transforms the inputs to its
interface into some lower-level abstraction. Clojure transforms Lisp
into JVM bytecode. Ansible transforms task definitions into shell
commands. In an ideal world, you wouldn’t have to understand the
internal model, but in reality it’s almost always helpful to
understand a tool's internal model because it gives you a unified
perspective on what might seem like confusing or contradictory
parts. When the double-helix model of DNA was discovered, for example,
it helped scientists make sense of higher-level phenonema. My point, of
course, is that this blog post is one of the greatest scientific
achievements of all time.

Tutorials often mix up a tool's external model and internal model in a
way that’s confusing for learners; it's helpful to be aware of this so
that you can easily identify when it's causing you frustration.

Spaced Repetition Helps You Remember

Spaced Repetition been proven to be one of the best ways to encode new
information in long-term memory. The idea is to quiz yourself at
ever-increasing time intervals to minimize memory decay using the
fewest number of repetitions. The Guardian wrote a
great introductory article.

Sleep and Exercise

Take care of your body! It's more than just a vehicle for your
brain. If you want to be able to stay focused and learn efficiently,
getting adequate sleep and exercise beats the pants off caffeine and
energy drinks.

More tips?

If you have any useful tips, please leave them in the comments! If
you'd like more excellent resources on learning to program, check out
the
Community Picks: Learn Programming,
a community-curated collection of the best books for learning
programming. It includes a wide array of subjects, including
introductory programming books, books on craftsmanship, and books on
soft skills and interviews.

Zen and the Art of Blowing Things Up

The Dark Knight came out at the same time that I was becoming intensely interested in Buddhism, and the movie struck me as a kind of extended Buddhist parable, with Bruce Wayne as an icon of the suffering that results from clinging and the Joker as a very fucked up enlightened being. Here are some of my main reasons: (warning: spoilers!)

Suffering and Impermanence

Buddhism teaches that suffering comes from constantly scrambling for stability in an unstable world. Nothing is permanent, but we strive mightily to hold on to the illusion of permanence, only to suffer when reality once again asserts itself.

In the scene where Batman is in the interrogation room beating the stuffing out of the Joker, the Joker just laughs and says, "You have nothing! Nothing to threaten me with. Nothing to do with all your strength." And that cuts to the heart of who the Batman is: a person whose reaction to the first shock of impermanence (his parents dying) was to dedicate his life to developing his strength so that he would never experience that pain again. The Batman's entire being is an exercise in doing just one... more... thing... to establish control.

I mean, who would want to live that life?

And here's the Joker, ready with the Zen slap: he tells Batman where to find Harvey and his girlfriend Rachel, giving him the illusion of control over who to save. And we know how that turned out: the Joker lied to the caped crusader, telling him Rachel was at one address, but actually Harvey was there. Batman races to save Rachel, only to find find Harvey instead. Nothing to do with all that strength.

This really hit home for me because, at the time, I was in a relationship with someone that had a chronic illness. I could try to do everything to make her life less painful and more enjoyable, but there was only so much I or anyone could do.

Natural Metaphors and Endless Desire

At one point, the Joker says, "I'm just a dog chasing cars, I wouldn't know what to do if I caught one." Or something like that.

This is a double whammy: "a dog chasing cars" is totally something a zen master would say, but besides that it expresses our human tendency to constantly strive for the next thing. We're all just dog chasing cars, not knowing what to do when catch one. You get a job: now you want a better job. You get a house: now you want a bigger house. Better to just say fuck it, put on some makeup, and start blowing up hospitals.

No Self

The Buddhist idea of "no self" is actually somewhat nuanced and I'm purposefully misrepresenting it here, but it would be difficult to argue that the Joker clings to any notion of self in the way that most of us non-enlightened psychopaths do. He gives multiple stories for how he got his scars, revealing the Buddhist truth that our sense of self is just a story we tell ourselves and that no one story is better than any other. He gives up his name and identity to become a fucking clown. And people really hate clowns!

Interconnectedness

One Buddhist teaching is the idea of interconnectedness: that we are all one, but not in the wavy-gravy way your high uncle used to ramble about. My take on the idea is that it's related to no self: you cannot point to a "you" that is separate from the world you exist in. It's a fallacy to believe your "you" is somehow different from the collection of experiences of the world, so that you are inextricably intertwined with the world. You can't point to one thing outside yourself and say "I am not this," because you are pointing at it you moron, and so in that moment who you are is a guy pointing at that thing, and life is the accumulation of such experiences of things outside of "you."

ANYWAY I am totally not your high uncle, OK? Point is, the idea of interconnectedness is meant to foster a sense of compassion. You can see that at the end of the movie when the Joker plays his hilarious "blow up the other boat" gag. Two boats full of strangers become enlightened, refusing to do harm to people they haven't even met.

The Dark Knight remains one of my favorite movies of all time, and it inspired a Batman obsession that took years to subside. (Ironically, obsession is a fairly un-Bhuddist behavior.) If you're a Batman fan too, check out Batman Community Picks for the best Batman books and movies. Or if you're interested in Buddhism, check out Buddhism Community Picks for the excellent books.

My Writing Process

The letter below is a response to an email from a Twitter friend. For
context: I've written some instructional material that's enjoyed
success online, including
Clojure for the Brave and True and
Clean Up Your Mess: A Guide to Visual Design for Everyone. Robert
asked me a few questions about my process.

Hi Robert!

You're in the unfortunate position of being my permission to write at
length about my creative process without seeming completely self
indulgent: someone's actually interested in listening to me describe
the chaos between my ears! Be prepared to read way more than you asked
for.

How do you decide who you are writing for?

This one’s key for me. When I’ve written in the past, I’ve struggled
with knowing how much or how little to say. I’m curious to learn how
you deal with that.

First: I've found it useful to follow the standard advice and pick an
individual or two and write for them. The argument goes that you'll
find it much easier to choose words and have a consistent tone, and
that's proven true for me.

When I wrote Clean Up Your Mess, I was one of three programmers
working in an office full of charming people who did business analysis
for management consultants, and I wrote the guide for a couple of them
that I had become friends with. These analysts produced powerpoint
decks by the barrel, and every single one was ugly. Similarly,
Clojure for the Brave and True started as a guide for a couple
coworkers that I was mentoring.

However, picking your audience is only half the process. The other
half is picking who the author is. I'm going to go on a bit of a
tangent to explain this.

When I started writing Clean Up Your Mess, my personal life was a
mess. In hopes of making it less messy I had been spending most of my
free time reading Buddhist literature and listening to lectures by
Thich Nhat Hanh and Pema Chodron (in fact, my site opens with a Thich
Nhat Hanh quote). I was spending a great deal of time meditating and
reflecting. (You're interested in Buddhism too, right?)

One of the ideas that really grabbed me was that our self-conceptions
are just stories we tell ourselves with no reality to them. I won't
belabor the point here; suffice to say that I found it useful and
powerful to be able to perceive my identity as a construct as
ephemeral as smoke.

Around the same time, I started taking improv comedy classes, where I
would spend two hours a week constantly coming up with new, unique
characters and living in them for a few minutes at a time. When you're
sold on the idea that your day-to-day you is itself just a
character, improv becomes way easier and more fun. It's also fun to do
this in real life - like, recently my wife and I were at the grocery
store and I decided my character was "Super Excited About Carrots
Guy", and I started shouting "OH MY GAWD JESS CARROTS! THEY HAVE
CARROTS JESS! CARROTS!" Everyone was laughing as they escorted me
out!

Anyway, during this time of buddhism and improv I was writing Clean
Up Your Mess and when I sat down to write I got to play with new
identities, new characters, and see which was the most fun. It became
much easier to find words and keep a consistent tone when I decided on
the author's character.

So, you mentioned not knowing how much or how little to write. Knowing
your character helps. In Clojure for the Brave and True, one of my
editors would sometimes whittle down sentences to the bone, leaving
only pure information; I would undo those edits with gusto, adding
back the shape and texture of the author's voice.

There's another aspect of writing instructional material, which is
that 95% of the time you can take more time to explain something. Even
if you're writing for experienced audiences, at least take the time to
explain acronyms and jargon.

How often and how much do you write?

Not enough and not enough! Writing was easiest, however, when I had a
consistent schedule of writing at least one hour every single day.

Let me tell you about my garage. My garage is a living f*cking
nightmare. It's like Cthulhu and the dread of the dark you had as a
child made disgusting love and produced a horde of horrors that died
in giant filthy clumps all over. At the beginning of the year I
suppressed my gag reflex long enough to clean it out with the goal of
setting up a photo studio, which I did, only to let it sit for three
months. So three months later I go in there to actually take photos
and guess what: it's a complete horror show, again!

If I had only spent time every day or even every week maintaining it,
I'm sure it would have been much easier to keep clean. If I had
cleaned it out, all those tiny insect monsters wouldn't have been able
to breed, there wouldn't have been two dozen spider egg sacs, etc.

When I don't write consistently it's like that. I have to crawl into
the nest of no one cares what you write and this time everyone's
going to hate it called my mind and clear it all out before I can
actually start writing.

When I write consistently, on the other hand, it's actually quite
pleasant!

One more note: much of the writing process is research. Every time I
find myself struggling to explain something, it's because I don't
understand it well enough. When that happens I put down the keyboard
and pull out the pen and notebook. I try to articulate the questions
that will lead me to understand what I want to explain. I read and I
draw diagrams and play with words until I feel like I understand the
concept.

Becoming aware that, for me, writer's block is a lack of
understanding, has helped me be much more productive.

Collaboration

Who does your drawings?

My wife! She's pretty great.

Do you have someone who edits for you?

I did when I worked with No Starch. They were awesome! Clojure for
the Brave and True is much, much better because they kept pushing me
to write more clearly and explain things more fully. For other
projects, I'll ask friends to look at drafts.

How much does it cost?

I paid my wife in carrots. OH MY GAWD JESS THEY HAVE CARROTS!

No Starch gets 85% of hard copy sales and 77.5% of ebook sales (or
something like that).

I hope this was helpful!

Understanding Recursion

During my Clojure training
I've found that recursion routinely trips people up. It definitely
tripped me up for a long time. I've tried to develop an explanation
that's clear and intuitive, so if you're still scratching your head
about recursion, read on!

A classic recursion example is calculating n factorial, which is n
multiplied by every natural number before n; 3 factorial is 6
(3 times 2 times 1), 4 factorial is 24, 5 factorial is 120.

The code snippet that follows is a typical implementation of
factorial; if you're reading this, then presumably it's confusing -
which is great! It means that I haven't written this article for
nothing.

function factorial(n) {
if (n == 1) {
return n;
} else {
return n * factorial(n - 1);
}
}

What makes this function recursive is that factorial calls
itself. That's also what makes the function tricky; the function calls
itself!?

We're used to functions calling other functions to get work
done. For example, this function uppercases a string and prepends
"Yo, " to it:

function yoShout(str){
return "Yo, " + str.toUpperCase();
}
yoShout("gimme a donut");
// "Yo, GIMME A DONUT"

In this tiny example, yoShout does its work by using the
toUpperCase function. It's easier to understand than a recursive
function because yoShout treats toUpperCase as a black-box
abstraction. You don't have to tax your brain by loading
toUpperCase's implementation details into your short-term memory.

Let's re-write factorial to use function calls this way, with
function's body calling another function in order to get its work
done. To calculate 3 factorial, you could write a series of
functions, factorial_1, factorial_2, and factorial_3, like this:

function factorial_1() {
return 1;
}

function factorial_2() {
return 2 * factorial_1();
}

function factorial_3() {
return 3 * factorial_2();
}

These functions feel safe and comfy. factorial_3 calls
factorial_2, something we're completely familiar with, and likewise
factorial_2 calls factorial_1. factorial_3 also does not care
how factorial_2, just like in the string example.

Unfortunately, these functions are also completely impractical; can
you imagine writing factorial_1000? The recursive implementation
doesn't have this problem.

My suggestion is to try seeing the recursive implementation from the
same perspective as the nonrecursive imiplementation. Here's the code
again:

function factorial(n) {
if (n == 1) {
return n;
} else {
return n * factorial(n - 1);
}
}

You can look at this and say, "Oh, if n isn't 1, then this function
does its work by calling some black-box function named factorial
with the argument n - 1." You can look at the call to factorial(n -
1) as a call to a completely different function - one that just
happens to have the same name and algorithm.

That's it! I hope it helps. If you've been confused by recursion, I'd
love to hear your feedback!

Timeless Programming Tools

I've been programming professionally for a dozen years. Some of what
I've learned is best forgotten (oh god, Cold Fusion), but there have
been many tools, techniques, and concepts that continue to be useful
every day. Here are mine; I'd love to hear what yours are so I can
experience both the joy of discovery and regret for not learning about
the tool earlier.

Relational Algebra / SQL

I feel lucky that, during my fourteenth summer, I apparently had no
friends and so had nothing better to do than try and slog through
a book on MySQL and the now-defunct mSQL. You
can see from the reviews that the book "is sketchy, incomplete, and
almost totally useless." But, it did introduce me to SQL and
databases. Soon after, I learned relational algebra (the theory
underlying RDBMSs) and that investment has been one of the best of my
life. I can't count the number of times a LEFT OUTER JOIN has saved
my bacon. Friends be damned!

Learning relational algebra provided the foundation I needed to move
easily from MySQL to Oracle and MS SQL Server when I joined
EnterpriseCo, and in general knowing how to interact with databases
without a framework or ORM helped me quickly advance career-wise. It's
why, at 20, I was able to land a contract building a custom site for
the city of Santa Fe, New Mexico, instead of just cobbling together
Wordpress and Drupal plugins.

If you come from Rails or some other framework that handles all the
database interaction for you, one of the best things you can do for
your career is to learn relational theory and SQL. Read
a book by C. J. Date.

The Unix Process Model

Understanding Unix processes helped me understand what's actually
happening when I run a program. It's also helped me understand what
exactly a web server is and what I'm doing when I write a web
application. The book Advanced Linux Programming has
a chapter on processes for free online. Actually,
the whole book is free.

When you don't know about processes, programming is much harder and
more mysterious. It's harder to understand performance, and it's
harder to understand how programs interact with each other. If you
ever feel a vague sense that you don't really get what's going when
you run the apps you write, learning the process model will go a long
way toward clearing things up.

Regular Expressions

Yeah, yeah, we've all heard the joke: "something something regular
expressions, then you have two problems." Personally, I don't get it,
because regular expressions are seriously badass. I remember going
through O'Reilly's big fat regex book while I worked from 11pm till
7am as a night auditor at a hotel when I was 18, and being blown away
at how powerful they are. To say that we programmers deal with text
all the time is so obvious, it's not even worth saying. Regular
expressions are an essential tool,
and here's where you can learn to use them.

Finite State Machines

Regular expressions are built as finite state machines. Here's
a great tutorial on FSMs
showing how to actually build a regular expression. It's extremely
cool!

I think FSMs are covered in computer science 101, but since I only
went to college for a year and even then I studied works written a
couple millennia before before the computer revolution, I didn't
actually learn about them until about six years ago. My colleagues and
I were having trouble with a mobile app - we needed the initialization
process to happen in a particular way, and the logic for ensuring that
was getting pretty tangled.

Once we took the time to learn about FSMs, though, it was easy to
express the process as a series of states and transitions. I've since
found that most tricky UI code can be improved this way. Just a couple
months ago I was having trouble building a typeahead element from
scratch with hoplon. Once I identified that the
difficulty was in keeping track of all the possible states, it only
took a few minutes drawing a state machine diagram and I was back on
track.

Emotion Management

In my personal life I'm constantly learning about and practicing ways
to manage emotions. This stems from both my personal aspiration to
improve the lives of others and from the completely selfish reason
that it helps me do good work. Emotion management is probably the most
important meta-skill you can develop. I mean, emotions are at the core
of who you are as a human being.

The book Non-Violent Communication is an
excellent resource for dealing with emotions. Also, my friend Alex
Harms recently wrote
a book specifically for technical people.

Those are my programming power tools - I hope you find them useful!

Brave Clojure: Become a Better Programmer

Next week week I'm re-launching
www.braveclojure.com as Brave
Clojure. The site will continue featuring Clojure for the Brave and
True, but I'm expanding its scope a bit. Instead of just housing the
book, the purpose of the site will be to help you and the people you
cherish become better programmers.

Like many other Clojurists, I fell in love with the language because
learning it made me a better programmer. I started learning it because
I was a bit bored and burnt out on the languages and tools I had been
using. Ruby, Javascript, Objective-C weren't radically different from
each other, and after using them for many years I felt like I was
stagnating.

But Clojure, with its radically different approach to computation (and
those exotic parentheses) drew me out of my programming funk and made
it fun to code again. It gave me new tools for thinking about
software, and a concomitant feeling that I had an unfair advantage
over my colleagues. So of course the subtitle of Clojure for the
Brave and True is learn the ultimate language and become a better
programmer.

And, four years since I first encountered Rich Hickey's fractal hair,
I still find Clojure to be an exceptional tool for becoming a better
programmer. This is because Clojure is a fantastic tool for exploring
programming concepts, and the talented community has created
exceptional libraries for such diverse approaches as
forward-chaining rules engines
and constraint programming and
logic programming, just to
name a few.

By learning a variety of programming concepts, you become a better
programmer because you're better able to recognize what kind of
problem you're working on and apply the right tools. To give just one
example: when I'm working on complex UI widgets my work is always made
easier if I model the widget using a finite state machine (FSM) - but
I had to actually spend time learning what FSMs are and how to use
them (those of you with comp sci degrees are probably chuckling at
this).

You may have started learning Clojure because you heard that
functional programming, Lisp, and immutability can make your life
easier, but the fun doesn't stop there! At Brave Clojure, I hope
you'll continue using the ultimate language to explore the vast and
infinitely fascinating world of computer programming with me.

The first topic we'll explore is parallel programming. Here's the
beginning of an ebook on the topic that I'm nearly finished with:

Yes, it would be fair for you to describe my startup as the Google of
palm reading. I don't know if you know this, but the fortune
telling industry has no quality control standards. There is no Six
Sigma of fortune telling. Sometimes it seems like these
people are just making things up!

We can do better. Did I say we? Yes! I want to you to embark on this
startup journey with me. Lucky you! My - our - plan is to disrupt
the field with an app that lets a user position their phone so that
the camera faces their palm and takes a picture, then predicts the
future. I call it: iFacePalm.

And, uh, there's parallel programming in it.

While I plan on spending most of my time helping you and all of your
closest friends, enemies, and even frenemies become better programmers
by writing about these big ideas, I'm also going to spend time
covering practical Clojure topics like deployment and debugging.

To make this happen, I'm working a (great) job only two days a week
and spending the rest of the time on Brave Clojure, so I'll need to
find a way to derive income from the site. I'm not sure yet what will
be free and what won't. My dream is to make all the content
available online for free. That's what gives me joy, and I think it's
the best way to have a positive impact. Ideally, I would be supported
in this endeavor through monthly donations, but I'll likely try
different approaches over time.

Finally, I've been putting together two sites to help you up your
game:
Open Source Clojure Projects,
a directory of active projects, and
Clojure Work, the first Clojure-only job
board. The latter is still in beta, and I'm going to officially
release it and move it to https://jobs.braveclojure.com next week. The
best way to get better at Clojure is to actually write Clojure, and
working on real projects (and even getting paid to do so) is a great
way to do that. Companies who buy job board ads will also be
supporting me in making more content - hooray!

If you get excited about learning more programming ideas, or learning
to develop Clojure programs better, or getting paid cold hard cash to
write your favorite language, then I cordially invite you to consider
signing up for my newsletter, where I'll announce every release:

Writing Clojure for the Brave and True was one of the most rewarding
experiences of my life in large part because I got to interact with so
many kind, thoughtful, intelligent people. I look forward to your
company as I embark on this new chapter of my journey!

A Directory of Open Source Clojure Projects

Every now and again I'll come across a post where someone asks about
getting involved in open source clojure projects. Because every
question deserves a single-page app dedicated to its answer, I made
Open Source Clojure Projects,
a directory of active projects welcoming new contributors.

Each project has (or should have) clear instructions on:

Developing the project locally
Running tests if there are tests
Contributing code (pull request? tests required?)
Contacting other devs - slack, mailing list, IRC, etc

Also, there's a "beginner-friendly" flag so that new Clojurists can
easily find projects with tasks that are appropriate for their skill
level.

So far, 22 projects have been added, which is awesome. A random
sampling:

milestones,
which generates the best possible schedule from a set of tasks

reverie, a
CMS for power users

Scheje, a
little Scheme on Top of Clojure (!!!)

system, a
friendly layer on top of Stuart Sierra's component library

If you have an open source project and want people to contribute,
please add it!

The Stack

A few folks have asked me about what tools I used to create the
site. I'll explain more below but briefly:

The backend is a weird, unholy mess of fantastic classic libraries
and sideshow code that I keep transplanting from one project to
another so that I can answer the question what kind of abomination
will this morph into this time?. The backend uses Github as a
database, though, and that's pretty neat.
The frontend uses re-frame and includes some form handling code that
I'm proud of and that's actually worth stealing.
Finally, there's also some code for deploying with Ansible that's
worth stealing.

In the time-honored fashion of technical blog posts everywhere, I'll
now "dive in" and elaborate on each of those bullet points.

Backend

For the backend, the site uses ring, compojure, liberator, http-kit,
and the weird set of tools that have accreted in my projects over the
years. Even though the code footprint for the backend is pretty small,
it's pretty idiosyncratic, containing handfuls of half-formed ideas I
keep noodling around with. Hopefully someday soon I'll be able to
really nail down my approach to backend dev and share it, because it
does allow me to write apps quickly.

One cool part of the site is that it uses a Github repo,
Open Source Projects,
as its database. Browse the projects directory and you can see every
project that's listed on the site stored as an EDN file. When the
backend starts up it reads from Github, and whenever someone posts or
edits a listing it writes the EDN file using Github's API.

The nice thing about this approach is that I didn't have to worry
about standing up a server or keeping backups. And it was just fun to
code. Here's the source
for using Github as a db - I definitely see potential in reusing this
approach for similar lightweight directory sites.

Frontend

re-frame has a delightful README
that gracefully introduces you to reactive programming. If you haven't
read it then stop reading this article and read that instead; I am
100% confident that it's a better use of your time.

re-frame is a joy to use, and it's become my go-to tool for frontend
projects. I've written a fair amount of code
for working with forms
and
submitting values. This
code forms a kind of nano-framework on top of re-frame, and it's
allowed me to speed up the development process from project to
project. Here's
an example form which uses the form helpers. If
you'd like for me to write a detailed explanation of this code, please
leave a comment letting me know :)

Boot deserves a special mention because it
makes it so easy to develop ClojureScript apps. With Boot you get live
reload (every time you change your frontend files, the changes are
automatically injected into the browser), giving you a
near-instantaneous feedback cycle that makes development much more
enjoyable. It's also easy to incorporate Sass
compilation - and your Sass files also get the live reload treatment.

If you're not using Boot to build your ClojureScript app then chances
are you're causing yourself undue suffering. And hey, maybe you're
into that? No judgment. But if you're not, then take the time to learn
it - it's a great investment in yourself as a Clojure developer.

Deployment

The
infrastructure
directory contains scripts for provisioning and deploying to a server
with Ansible. It also has a
Vagrantfile so that you can stand up a
test server locally; just run vagrant up to both create the virtual
machine and provision it. To deploy, cd into the infrastructure
directory and run ./build.sh && ./deploy-dev.sh. Those two shell
scripts are extemely simple, so take a look at them to see what's
involved in deployment.

The Ansible configuration is almost completely reusable. You can check
it out in infrastructure/ansible. The roles directory in
particular has Ansible tasks for creating an nginx server that can run
Clojure app. It also includes an Upstart script for controlling
startup/shutdown of your Clojure app (assuming you use Ubuntu).

The Future

Eventually, I'd like to add more tools for project discovery and for
showing project stats. @yogthos, for example, suggested incorporating
a project's stats from github so that users can see how active and
popular each project is, information that can help them decide which
project to contribute to.

Also, I'm in the process of revamping
braveclojure.com to include more resources
for Clojure programmers. The site hovers around 3rd or 4th Google
result for "Clojure", just below clojure.org and the Wikipedia
entry. So, it's kind of popular I guess. My hope is that by featuring
Open Source Clojure Projects, I'll help more new developers get
involved and also help project maintainers squash some tickets.

Not a Superpredator

I'm the son of a Vietnamese emigrant. I grew up in the South, where I
experienced and witnessed racism. I have black step-siblings and my
wife is black (naturally, so are my in-laws). As such, I try to
educate myself about race issues in America. I don't share my thoughts
much publicly but do want to write more on the subject. In the mean
time, here's a (slight edited) exchange I had with a friend on
Facebook about a recent event,
Ashley Williams's protest during a Hillary Clinton speech.

Me: A room full of white people at a $500/person Hillary Clinton event
tells young black woman she's being rude and inappropriate for
demanding Hillary Clinton explain why she supported the system that
destroyed countless black and latino lives. I think that accurately
sums up the prevailing attitude toward race in America: black people,
please don't cause me mild discomfort by bringing my attention to our
society's racial caste system. It's impolite.

Friend: Wow. This thing seems like... a really, really huge
distortion. She [Hillary] actually said: "You’re the first person to
do that, **and I’m happy to address it."** She offered to talk
about it. But the demonstrator wasn't there to talk, and wasn't in
listening mode. Second, what she said, 20 years ago, is that she
thought 'SuperPredators' exist and should be dealt with. It was a
popular idea at the time. She did not say that they were one color
or another: just that they exist and should be contained. WTF ??? It's
indisputable that America needs to deal with the legacy of excessive
force and excessive incarceration against people of color. But if we
make it sound like any talk about incarcerating anybody, ever, must be
inherently racist... we've already lost the battle to have serious
conversations about what non-racially-biased policing would look like.

Me: Hey man! Thanks for sharing your thoughts. Here are more of
mine, and I hope they don't come across as combative - I really
appreciate getting to talk about this stuff with someone as
thoughtful as you.

I wasn't actually commenting on Hillary's reaction, but the reactions
of those in the (almost entirely white) room with her: people hissing
at her and telling her she's being rude and inappropriate. That's what
I'm referring to when I say the video reflects the prevailing attitude
toward race. To me, it's as if the message toward the BLM activist
[note: I've since learned she actually isn't with Black Lives Matter]
is, "Excuse me, but you're interrupting the event we paid $500 to
see. Please take your complaints about the racial caste system, which
doesn't negatively affect me, elsewhere."

And it is always "rude" or "inappropriate" or "inconvenient" for those
topics to be discussed. When Beyonce dances at the Super Bowl, people
lose their minds: they're just trying to enjoy football, they don't
want to have to think about the lives of people who aren't like them!
When another black man or woman gets murdered by police and people
take to the streets demanding change, they're somehow offensive and
ungrateful for criticizing law enforcement.

Most often, the reaction is to denounce the method and ignore the
message.

Last year, Bernie Sanders visited Liberty University, and David
Nasser, a university VP, got some of the biggest applause of the event
for saying that the problem with the criminal justice system is a "sin
issue, not a skin issue" - meaning that it was not a political issue,
but one better addressed by encouraging people to be better or
something. Same message: stop speaking up, black people.

So that's what I was meaning to communicate originally. Here are my
thoughts on Hillary.

I'm not sure if you're familiar with "dog-whistle" politics, using
coded language to court the racist vote. At the time, the Democratic
party was trying to win back the vote of poor, Southern whites who'd
switched to the Republican party thanks to the republicans' Southern
Strategy, an effort
to win those votes by appealing to racist fears and biases.

Anyway, Democrats had the idea, "hey maybe if we also play on those
racial fears, we'll get those votes back!" And that's what they
did. When Hillary and other political/media figures talked about
"superpredators", it was generally understood to mean "urban" people,
and that was generally understood to mean black people. So when the
demonstrator brings this up, she's not saying that "any talk about
incarcerating anybody, ever" is inherently racist. She had reasons for
singling out that phrase and using it to call attention to Hillary's
record. Some might disagree, but I think her reasons were good ones.

And even if "superpredator" wasn't meant that way, the result was
still the crushingly unjust system we have today - and Hillary's
support helped build that system. Using terms like "superpredator" and
"bring them to heel" to accidentally perpetuate a racial caste
system might even be worse.

Michelle Alexander, author of The New Jim Crow,
writes about it persuasively. The
New Jim Crow goes into even greater detail about how mass
incarceration and the criminal justice system are tools for racial
control, even if many of the actors in the system (police, lawyers)
aren't racist.

I'm a Bernie fan, and I don't have rabid Hillary hatred. If it comes
down to Hillary versus whichever clown the Republican party decides
on, I will vote for Hillary. I respect her vast experience and think
she would do a decent job at pushing for progress. When it comes to
racial justice, though, I think Bernie has a much better record.

Most of all, I'm an Ashley Williams fan. It takes courage to do what
she did. Maybe she wasn't in listening mode (it's hard for me to
really articulate my thoughts on this. She did give a good pause so
that HC could speak. And anyway the power was so asymetrical that I
don't know how to think about it. How do you really try to talk when
you're being carted off by security?). But I'm glad she did it - it
brought attention to an under-reported problem, and Hillary did in
fact end up apologizing. And no damage was done, except perhaps to the
sense of propriety of a few stuffy rich people.