Will Larson's Blog, page 34
September 8, 2019
Organizing team offsites.
Last week, the team I support had our quarterly offsite. I’ve been running team offsites more and more frequently over the past few years, and have finally been able to nail down an approach that consistently feels like an essential, effective use of time.
My rules for effective offsites are:
Start with trust and safety. A good offsite starts with a group of folks who trust each other and feel safe bringing literal and intellectual honesty into the room. If you’re not bringing that team into...
August 31, 2019
Rewriting the Yahoo! BOSS Datahub.
In 2008, my approach to blogging was concocting an explosion of content on breaking technologies and hoping that would translate into notoriety, page views, or at least a job beyond teaching English. I attempted this approach to various degrees with PyObjC, Processing.js, Why the Lucky Stiff’s Shoes, and a few others.
This worked fairly well–density, novelty and recency are the necter that tempt the search relevancy gods–and eventually nudged open the door to my first job in techn...
August 17, 2019
Describing fault domains.
Fault domains are one of the most useful concepts I've found in architecting reliable systems, and don't get enough attention. If you want to make your software predictably reliable, including measuring your reliability risk, then it's an extremely useful concept to spend some time with.
A fault domain is a collection of functionality, services or components with a shared single point of failure. A fault level is a set of one or more fault domains serving the same purpose, for example if you...
August 11, 2019
Distributed systems vocabulary.
I've spent a fair amount of time recently discussing how distributed systems fail and strategies for preventing those failures, and the lack of a shared vocabulary poses a significant barrier to high-bandwidth communication.
Given some murky personal plans to write more in this area for the next bit, I've written up the vocabulary I've found most effective. Hopefully this will be useful in its own right, but at worst I'll use it to avoid repeating boilerplate definitions going forward!
Want...
July 28, 2019
Reclaim unreasonable software.
Big Ball of Mud was published twenty years ago and rings just as true today: the most prominent architecture in successful, growth-stage companies is non-architecture. Crisp patterns are slowly overgrown by the chaotic tendrils of quick fixes, and productivity creeps towards zero.
After a couple years wallowing at zero, many companies turn towards The Grand Migration pattern and rewrite their software. These rewrites are quite challenging and often fall.
An alternative path, which I wrote abo...
July 14, 2019
What's the inverse of literate programming.
One of my favorite ideas in programming that hasn't ever quite worked is literate programing. Literate programming explores how the act of writing and reading software might work if we blended code and documentation–the how and the why–much more closely, usually in the same file.
As I reflect back on writing my book, I wish I had tools to do the opposite: powerful, easily extensible tools to write books. I guess we could call it "programmable literacy", if we had to manufacture a...
July 7, 2019
Head in the clouds.
When I wrote about the public cloud evolving the role of infrastructure engineering, I sort of imagined that the precursor question–should we run our infrastructure on the public cloud?–was already quite settled. Unexpectedly, it's a discussion that I find myself having more rather than less frequently each year, so I've taken some time to structure and document my thinking.
In short: run on the public cloud unless (1) it prevents you from executing on your core competency or (2)...
July 3, 2019
Don't follow the sun.
When I speak with engineering leaders, I sometimes get asked to endorse an underway plan to spin up a “follow the sun” on-call rotation. Instead of one team taking pages for the full day, they'll split the load into 2 twelve hour shifts or 3 eight hour shifts.
My advice is not what folks anticipate: please don’t.
"Follow the sun" model is an evolutionary dead end for engineering on-call rotations. Having run the “follow the sun” on-call rotation for several years at Uber, I’ve come to believe...
June 30, 2019
Notes on Escaping the Build Trap
Last year while thinking about product management in infrastructure I read Cagan's Inspired: How to Create Products Customers Love, which I thought was a solid book, roughly summarized as "use rapid experimentation to guide product innovation."
It's apparently the time of year when I read about product management, and having heard quite a bit of praise for Melissa Perri's Escaping the Build Trap, I decided to give it a read and write up book notes.
These are some core quotes from the book, al...
June 26, 2019
Fostering program engagement.
Some of my favorite pieces to write are those that end up being interesting to a pretty small audience, but tap into a central nerve for that small audience. So far, Programs: tips for owning the unownable, has been in that category. Most folks don't engage much with leading programs, but folks who are have feelings and thoughts.
One particularly good email came in from someone creating a new program, who was curious about advice for a common program challenge:
I joined my current company as...
