Jason Fried's Blog, page 3

On Apple’s monopoly power to destroy HEY

This statement was delivered to the democratic side of the House Antitrust Subcommittee upon invitation on July 17, 2020 in the committee’s preparation for the forthcoming July 27, 2020 showdown with the Big Tech CEOs.

Two years ago, we got the audacious idea to take on Google, Microsoft, and Verizon to provide a new, fresh alternative to their email services. It’s been about 16 years since people were last excited about getting a new email address – Gmail was introduced in 2004 – and frankly, it shows. These legacy services have been virtually devoid of innovation for over a decade.

And why wouldn’t they be! They’ve already captured the market. Between Gmail, owned by Google, Hotmail/Outlook, owned by Microsoft, and AOL Mail/Yahoo Mail, owned by Verizon, you have about 85% of the US email market captured by three players. And out of that, Gmail alone is about 55 percentage points.

But! Despite this near-total capture by big tech, the underlying protocol of email is still just barely free and open. You don’t have to ask anyone’s permission to make a new email service. (Or so we thought.)

Fast forward millions of dollars in investment to June 15, 2020, which was the day we opened the doors to our new email service, and were met by every entrepreneur’s dream: amazing reviews and customers tripping over themselves to signup and pay for our product.

But the celebration didn’t last long. That same day, we also received a rejection letter from Apple’s App Store, saying that the HEY application they had just approved a few days earlier, was actually not in compliance with their rules after all, and unless we started using their payment processing system, along with the 30% cut of revenue it would entail, they’d kick us out of the store.

The first reaction was simply disbelief. We’ve been in Apple’s App Store with our Basecamp application, which also sells service subscriptions on the web, and simply has an iOS application that allows someone to access that subscription, since around 2012. This was not our first time at the rodeo!

We had studied Apple’s written rules, the precedence they’ve set, and were well-aware of all their unwritten rules too. Gmail, for example, also sells subscription email services and has the #1 Productivity app in the App Store. But they do not pay Apple 30% of revenues nor offer in-app payment. So we thought this was simply a mistake, and tried to appeal through all the backchannels.

None of that worked. Apple doubled down the very next day on us either paying them 30% or face eviction from the App Store. So here we are, on day two of a massively successful product launch, which was supposed to be about fighting THE OTHER big tech monopolist in our orbit, Google, and we’re suddenly in a battle for our very existence over Apple’s threat to kick us out of the App Store.

After exhausting all our modes of appeal, I went bananas on twitter in a thread that went viral, and which prompted dozens of dozens of other developers to reach out to me in private with similar tales, the press to take a keen interest, and customers to be outraged at Apple’s shakedown tactics.

So let’s pause for a moment here in this story. Why is what Apple’s doing a problem? Couldn’t we just take our service somewhere else?

In a word, no. Apple is unlike any other platform when it comes to the North American market for premium services like HEY. As of today, 90% of everyone who pays for our service uses at least one Apple device. 3/4 of all paying customers have the iOS HEY app that Apple was threatening to evict installed on their phone.

Apple isn’t just a dominant player in this market, they are the market. Without access to the iPhone, we’re dead. And even customers who’d ALREADY signed up for our new service told us as much: Unless we could guarantee that our app would be available and be updated, they just couldn’t commit to becoming or staying customers.

So those are the stakes: Apple controls who can and cannot compete in virtually all sectors of the digital economy through the monopoly power they wield capriciously with the App Store. It is simply not possible to even get the chance to take on the likes of Google, Microsoft, and Verizon with a new email service without begging Apple for permission to do so.

We were sitting on a rocket ship of a successful new service, eager to hire more people (we counted that we’d need to hire 12 more people to deal with the new business!), but were at the same time essentially paralyzed from making any investments, because what if Apple wouldn’t give us the permission to exist?

It’s worth noting here that we are already paying Apple for the privilege of having access to the App Store. All developers must pay $99/year for a developer license. Apple brags of having millions of developers, so they’re in essence already making hundreds of millions of dollars, just in licensing fees. Nobody is asking for a free ride here! If it costs Apple more than hundreds of millions of dollars to run the App Store, they can raise their prices. We’d gladly pay $199/year for a developer license.

But what Apple is asking for is a cut of revenues, at “highway robbery rates”, and it’s simply absurd. Imagine if the telcos demanded a cut of company revenues, since they provide the phone line that connects customers, in the heyday of their monopoly? Imagine if the railroads demanded a cut of company revenues from the goods shipped in the heyday of their monopoly?

Apple’s customers have already paid Apple upwards of $1400 to buy their iPhone. This is why Apple is a 1.67T company as of today! They are the most profitable technology company the world has ever seen. They already charge developers licensing fees that amount to hundreds of millions of dollars in order to reach these iPhone users. But now they also want to force the entire digital economy to hand over 30% of revenues? Absurd is one word you could use, illegal should be another.

Anyway, back to our tale of woe. After a two-week campaign, fighting on twitter, in the media, and seeing customers support our cause in the thousands, Apple backed off. But only by an inch. We escaped the 30% cut, but they forced us to come up with a new free, temporary email service, such that the app would “do something” when you download it. That solution came from a new, unwritten rule that Phil Schiller, the Apple VP responsible for the App Store, made up on the fly in interview with the press.

Never mind the fact that there are scores of apps that follow the path we had originally taken, where you can’t use the app unless you login with an existing subscription bought elsewhere. Including Netflix, most banks, and one of Apple’s own apps. Someone compiled an inventory of examples at https://youdownloadtheappanditdoesntwork.com.

So Apple, who otherwise purports to being this champion of privacy, is telling a new privacy-focused, for-pay email service that we must offer a free tier to be allowed in the App Store. I’m sure this committee is well aware of how companies have been monetizing free services on the internet. It’s usually with invasive ads, sale of data, or other privacy-intrusive tactics.

A company like Facebook, that makes tens of billions of dollars every year from their iOS app, simply has to pay Apple the $99/year developer license. But our HEY service would have to pay Apple $300,000/year, if we made but a million dollars? It doesn’t take a degree in formal logic to spot the contradiction.

Apple should not be able to restrict software innovation in the US. They should not be allowed to tell a service like HEY whether we can run a paid service or have to offer a free tier. Even our story with HEY, despite that we won some relief in the court of public opinion, has become a cautionary tale. What other small software company is going to invest years of effort and millions of dollars in a new service, if Apple might kill them dead on their birthday. This is how you stifle entrepreneurship, depress innovation, and prevent the next generation of makers from ever having a chance to compete.

I’ve focused on Apple in this example, because that’s the surprise adversary we faced. But the story could just as well had been about Google, if our app had been of a different sort or in a different market. Google’s Play Store policies are essentially the same as Apple’s, although they’ve wielded them slightly less aggressively so far. But make no mistake, if Apple gets off scot-free, Google is surely coming for their 30% cut of the entire digital economy as well.

And this is really the essence of this investigation, isn’t it. In everything from software to advertising to books and beyond, the big four tech companies have captured insufferable monopolies that they’re using to distort and shakedown markets, prevent new companies from succeeding, and ensure they never actually have to compete on the merits of innovation.

It’s indeed insufferable. And congress must act.

Specifically for Apple, our plea is simple: Force Apple to give us the choice of payment processing, and the freedom to tell customers. If Apple faced competition for their 30% cut, it would not be 30% for very long. Customers would win by seeing lower prices, be able to purchase subscriptions directly from companies, and Apple’s competitors would not be unfairly disadvantaged.

This is the essence of the two EU investigations that, as irony would have it, were announced the very same week that Apple threatened to kill our new business. It won’t solve everything, but would go a very long way to restoring basic competition.

See also my past testimony in front of the House Antitrust Subcommittee, which was delivered before HEY had even launched, and addresses the same issues around the App Store.

Basecamp’s Ops Team is Hiring

Basecamp is hiring three new System Administrators for our Operations team to help us deliver fast and reliable applications, like Basecamp and our new email service HEY. Our infrastructure exists both in colocated data centers and in the cloud, and you’ll be working alongside our existing team of Blake, Eron, John, Matt, Matthew, Nathan, and Troy.

As you might gather from the names, our operations team today is not nearly as diverse as we’d like it to be, or as the rest of the company. We therefore strongly encourage candidates of all different backgrounds to apply. Basecamp is committed to building an inclusive, supportive place for you to do the best and most rewarding work of your career. We are an equal-opportunity employer and are committed to building a company that embraces and celebrates diversity and inclusion.

About the work
The job of operating fast and reliable systems is mainly about making life as boring as possible by considering and mitigating all the things that could go wrong before they do. Operations shine the brightest when there’s nothing to see – systems just run. But getting there is both difficult to accomplish, and hard to sustain.

It’s also impossible to get perfect all the time. So part of the work includes being available for on-call duty, which may include getting woken up in the middle of the night, when it’s your turn to be first responder. We rotate that responsibility on a regular schedule.

Our standard stack includes MySQL (both Aurora on AWS and on-prem installations), Redis, ElasticSearch, S3, Ruby on Rails, postfix, Kubernetes, and local installations.

Here are some things we’ve worked on recently that might give you a better sense of what you’ll be doing day to day:

Integrate and optimize rspamd for inbound and outbound mail pipelinesModify and extend Terraform configurations for managing AWS resourcesPractice incident response procedures including service failoversProvision and load test both on-premise and cloud-based infrastructureManage database schema changes using pt-online-schema-changeUpgrade production Kubernetes clusters to a new releaseMigrate workloads from cloud to onprem or vice-versaDeploy and operate database clusters built with Vitess and MySQL

You’ll be working hand-in-hand with a team that is happy to answer questions and share their collective wisdom at each step of the way. And your experience and background will join ours! We have worlds to learn and we’re eager to do it together. You can expect to teach us, coach us, and expect the same in return.

About you
We’re looking for experienced candidates only for this role. You should have been doing this work before in a professional setting, but you also won’t know how all our systems work on day one, and we don’t expect you to. Nobody hits the ground running.

You should be familiar with our standard stack, completely comfortable with the Unix toolchain, basic networking concepts, and be at least a passable programmer, capable of writing or amending existing systems in Go or Ruby.

If you have experience with email services in particular, that’s a plus. We’re new to operating a full-scale email service with HEY, so if you can bring experience and fresh ideas, we’re eager to learn. But this is not a requirement.

You might have a CS or engineering degree. You might not. That’s not what we’re looking for. We care about what you can do and how you do it, not about how you got here. A strong track record of conscientious, thoughtful work speaks volumes.

We value Managers of One. We appreciate people who can take a stand yet commit even when they disagree. We subject ideas to rigorous consideration and challenge each other, but all remember that we’re here for the same purpose: to do good work together. That comes with direct feedback, openness to each others’ experience, and willingness to show up for each other as well as for the technical work at hand. We’re in this for the long term.

You can expect a mindful onboarding process with ramp-up and time to learn. You can expect a team that listens, and to be heard. You can expect to give and provide direct feedback. You can expect to be counted on. You can expect to do your best work and build a career here. We build our apps, our teams, and our company for the long haul, with a strong eye to other teams’ effectiveness & happiness, from design to support and operations. A healthy, resilient app starts with a healthy, resilient organization.

This is a remote job. You’re free to work where you work best: home office, co-working space, coffeeshops. We don’t even have an office at the moment! However, given our on-call responsibilities, we’re specifically looking for someone to complement our existing timezone coverage.

We will therefore not be looking for additional team members in the central or eastern timezones of the US. We’ll consider mountain time, would be happy with pacific time, and would be thrilled with candidates in Hawaii or Europe. But we also need a reasonable overlap during a normal workday with the central timezone, so we are not considering candidates from, say, India for this role.

Applicants from outside of the US will be offered a contractor role on comparable terms and equal pay with our domestic employees.

Pay and benefits
Basecamp pays in the top 10% of the industry based on San Francisco rates. Same position, same pay, no matter where you live. The salary for this position is $134,257 (for Systems Admin) or $167,962 (for Senior Systems Admin). We assess seniority relative to the team at Basecamp during the interviewing process.

Benefits at Basecamp are all about helping you lead a healthy life outside of work. That starts with working no more than 40 hours a week on a regular basis and getting 8+ hours of sleep a night. Our workplace and our benefits are designed to support a sustainable, healthy relationship with your work.

We offer fully-paid parental leave. We work 4-day weeks through the summer (Northern Hemisphere) and take a one-month sabbatical every three years. We subsidize co-working, home offices, and continuing education, whether professional or hobbyist. We match your charitable contributions. All on a foundation of top-shelf health insurance and a retirement plan with a generous match. See the full list.

How to apply
Please send an application that speaks directly to this position. Show us your role in Basecamp’s future and Basecamp’s role in yours. Address some of the work we do. Introduce yourself as a colleague. Give it a deep think and take this at your speed. We’re giving you our full attention.

We’re accepting applications until Sunday, August 2, 2020, at 7PM US-Central time. There’s no benefit to filing early or writing a novel. Keep it sharp, short, and get across what matters to you.

Next: We expect to take a few weeks to review all applications. You’ll hear from us by Friday, August 21, about advancement to a take-home technical exercise. This test is offered to no more than 20 candidates out of the usually hundreds (or in some cases, more than a thousand!) people who apply. You should expect to spend upwards of a full day (but no more) completing this test. Please consider whether you’d be willing to set aside time for that before applying.

After the written test, the top candidates will proceed to an interview. Our interviews are one hour, all remote, with your future colleagues, on your schedule. We’ll talk through your background, your approach to work, and dive into your technical knowledge. No gotchas, brainteasers, or whiteboards. 

We aim to make an offer by the end of September with a start date in October.

Note: We’re unable to offer feedback during most of the screening process. We tend to see over 1000 applications for our technical roles. We don’t have the bandwidth to offer personalized feedback to candidates that do not proceed to the interview round.

This is a demanding application process, and significant career move to consider. We appreciate the generosity and courage that go into giving us that consideration. Our conversation starts with this post. We look forward to hearing from you!

Apply here

How we achieve “simple design” for Basecamp and HEY

Yesterday I got an email asking how we achieve simple designs for Basecamp and HEY, so I hastily tweeted a screenshot of my answer, and a lot of people responded to it. A few folks pointed out that my screenshot was illegible for blind users, so I’m reposting it here in full text, with a bit of additional commentary below.

At a high level it boils down to a handful of foundational principles that affect the decisions we make:

Always choosing clarity over being slick or fancy. Internally we call this “Fisher Price” design. We aim to make the UI totally obvious and self explanatory, by keeping individual screens simple, showing only one focused thing at a time, and so on. Good product design eliminates the need for an instruction manual!Preferring good copywriting, and taking the time and space to explain things with words, instead of making minimalist UIs with lots of unlabeled buttons, etc. (Although we’re still guilty of having a few of those.)Prioritizing respectful interfaces that don’t overwhelm or try to nag the user into certain behaviors. We intentionally don’t include things like notification counts/badges, 3-column designs, and such unless we absolutely can’t avoid them. We don’t like the idea of having “sticky” interfaces—we want our customers to use our products to get the job done, and then go do something else. That makes the whole design approach more peaceful in general.Having a strong editorial sensibility, and knowing when to split complex concepts into simpler individual parts. This one is more of an art than a science, but we have a good instinct for breaking down problems until they can be easily understood in simple UI flows.

There’s one other thing that’s important for simple design. It’s not merely a matter of having clear or basic-looking interfaces. (It’s easy to make a simple UI that doesn’t really do much.)

The magic combo is having simple interfaces paired with powerful capabilities below the surface. HEY looks simple, and it’s straightforward to use, but it’s backed by some deeply considered opinions, logic, and machinery that empowers people who use it to be better at email than they were before—and with less effort! The interface itself is simple, but the thinking and the system behind it is extremely complex. The product is valuable because it saves people time and anxiety dealing with the terrible email mess that they had just learned to live with.

For more on this line of thinking, we highly recommend Kathy Sierra’s Badass: Making Users Awesome.

(P.S. many thanks to @waynedixon and @redcrew for calling me out on the inaccessible screenshot tweet. Appreciate that.)

The evolution of HEY: from humble beginnings to a multi-platform email service

[image error]

Two weeks ago we released HEY into the world, the culmination of 2+ years of explorations and intensely focused work.

It goes without saying, but I’ll say it anyway: inventing a new product from scratch is one hell of a challenge. It’s the toughest thing you’ll ever do as a product team. There are a million reasons why it won’t work, and zero guarantees that it will, so the whole project is a massive gamble. You just have buckle up and trust that you’ll figure things out.

Since HEY made a big splash on arrival, I thought it’d be fun to share the backstory of how we ended up reinventing email. Because we certainly didn’t start by wanting to reinvent email. (That sounds hard and intractable!)

For the past several years, even pre-dating the work on HEY, we noticed some ongoing communications problems at our company. They all revolved around external email: things like going back and forth with lawyers, working with benefits providers, dealing with press outlets, accountants, and stuff like that.

It was too hard to keep the right people in the loop, because we’d have lots of email threads scattered around with different participants, and none of them had the complete story. It was too hard to get caught up on a project that someone else was working on with a vendor. And too hard to hand work off when someone went on vacation or parental leave. At best, you’d end up with a steaming hot pile of forwarded emails, and it was all a chaotic mess.

This was in total contrast to our well-tended internal communications, which all lives tidily in Basecamp (the app), and suffers none of those problems. But Basecamp is mainly a company tool, and it’s not so reasonable to ask our miscellaneous vendors to join Basecamp just to communicate with us. So what’s the equivalent of Basecamp for the outside world?

Well, it didn’t exist! So we began thinking about how we might solve these problems with a new product.

At first, we were considering making a successor to our older CRM product, Highrise. Highrise was our first take on the external communications world, but it doesn’t do nearly enough to cover all the scenarios we wanted to solve.

And thus, HEY started its life as a prototype for a possible Highrise 2. It was originally focused around work problems—not personal email problems.

In May of 2018, Jason and I kicked off the first work on a prototype, built in Jekyll, which was code-named Haystack. We were intent on addressing real-life situations right away, so I asked our Head of People Ops, Andrea, to find the gnarliest vendor email threads she’d been involved with recently. She forwarded all the emails to me, along with any internal chats she had about those emails. Using that source material, I painstakingly reconstructed all the conversations into seed data for our prototype. That way we’d have some real stuff to judge our ideas against.

We started by designing how these internal/external threads might look and work, then gradually worked our way out to the Inbox view, and finally filled in various supplemental ideas around the edges. We focused specifically on email pain points, and discarded any notion of how email “usually works” in favor of the way we would ideally want it to work, while also considering what we could realistically foresee building.

After a few months of prototyping, it looked like this.

[image error]Thread view[image error]Thread with comments[image error]Early Inbox[image error]Early Screener (nee Bouncer)[image error]All Files view[image error]Contact page conceptPre-production Haystack Prototype

Having explored lots of different examples and core ideas (including the nascent version of The Screener, the All Files view, and contact pages), we felt like we had something promising.

None of it really worked, there were still a million unknowns, and loads of features we hadn’t figured out yet, but there was one thing we were sure about—it had evolved far beyond just a better Highrise. If we were going to build it, it needed to become a complete email service. And if it was going to be an email service, it needed to have a personal version, of course!

Phase 2: Building it

By early 2019 we started the process of making HEY real. At first we used the prototype’s general skeleton as a foundation to start wiring things up, but we continued to hone the app’s visual design. Our brilliant marketing designer Adam joined up, and he helped us shed all the experimental cruft from the original design, paring things back and making the whole thing more minimal.

Adam also developed the original color palettes and menu gradients that became iconic—those bright pops of color are sprinkled throughout and give the app a lot of friendly personality without being too overwhelming.

[image error]Refreshed Thread[image error]Early Stacks concept (this became Labels)[image error]Stripped down Inbox
[image error]Menu styles[image error]Early menu color theme[image error]Basic Screener2019-era Haystack: A mix of styles, ideas, experiments, and skeletal minimalism

Phase 3: Tightening up, adding features, and doing all the hard work

Designing with a team where everyone has autonomy is a bit like playing in a band. Someone seeds an idea, and then someone else builds on it and takes it somewhere else, then those changes get promoted into little systems, and over time the best ideas persist through all the changes. Harmonies prevail.

You can see that taking place in the screenshots above, but it really picked up steam from mid-2019 on, when the whole the company started working in earnest. For a while there was a lot of tug-of-war design and development between the web and mobile teams, where all of us bent and stretched the system for what we needed on our respective platforms. Since HEY is a hybrid web app, we ended up building a ton of new tech to optimize for UI patterns that would work equally well on a phone or a laptop (we’ll be sharing a lot more about that soon.) Conor led the way on a lot of those efforts.

By early 2020 we had worked out most of the fundamentals. We had Reply Later and Set Aside, and we had The Feed, which at that time was called “Slowbox”.

[image error]The Feed, formerly known as Slowbox[image error]Imbox trays (before they stacked up)[image error]Early iOS app designUnrefined fundamentals

From then on, the remaining work was a recurring cycle of polishing what we already had, and using the product ourselves to figure out what was still missing. For example, in January I noticed I was having trouble filing the gobs of junky capitalism emails I constantly receive, so I pitched the idea of a special place just for that, and the Paper Trail was born.

[image error]Pitch for the Paper Trail (originally Baconbox

Towards carbon negativity

Humans have been pumping greenhouse gases into Earth’s atmosphere at an unsustainable rate. It’s on us to reverse course as quickly as possible to stay below the tipping point of 1.5℃ global warming. Without action, the future is beyond bleak.

At Basecamp, we’re committing to becoming carbon negative for our cumulative history and moving forward.

The first step of that commitment is understanding the size of our carbon footprint. We’ve just completed our first carbon footprinting exercise as a company and are publishing our results. By sharing our approach, we hope to make it easier for other companies to also join the collective mission to carbon negativity. 

In 2019, Basecamp’s estimated carbon footprint was ~820 tCO2e. Here’s how that breaks down:

Running spot instances effectively with Amazon EKS

Since we started working on HEY, one of the things that I’ve been a big proponent of was keeping as much of the app-side compute infrastructure on spot instances as possible (front-end and async job processing; excluding the database, Redis, and Elasticsearch). Coming out of our first two weeks running the app with a real production traffic load, we’re sitting at ~90% of our compute running on spot instances.

Especially around the launch of a new product, you typically don’t know what traffic and load levels are going to look like, making purchases of reserved instances or savings plans a risky proposition. Spot instances give us the ability to get the compute we need at an even deeper discount than a 1-year RI or savings plan rate would, without the same commitment. Combine the price with seamless integration with auto-scaling groups and it makes them a no-brainer for most of our workloads.

The big catch with spot instances? AWS can take them back from you with a two-minute notice.

Spot-appropriate workloads

Opting to run the bulk of your workloads on spot only works well if those workloads can handle being torn down and recreated with ease (in our case, via Kubernetes pods). What does “with ease” mean though? For us it means without causing exceptions (either customer-facing or internally) or job failures.

For HEY, we are able to get away with running the entire front-end stack (OpenResty, Puma) and the bulk of our async background jobs (Resque) on spot.

Enduring workloads

What doesn’t fit well on spot instances? For HEY, we’ve found three main categories: long-running jobs, Redis instances for branch deploys, and anything requiring a persistent volume (like certain pieces of the mail pipeline). We put these special cases on what we refer to as the “enduring” nodegroup that uses regular, on-demand instances (and probably RIs or savings plans in the future).

Let’s take a look at each:

Jobs that have the possibility of running for more than a minute-and-a-half to two-minutes are an automatic push over to the enduring node-group. Currently for HEY this is just account exports.Redis is a primary infrastructure component for HEY. We use it for a bunch of things — view caching, Resque, inbound mail pipeline info storage, etc. HEY has a dynamic branch deploy system that deploys any branch in GitHub to a unique hostname, with each of those branch deploys needing their own Redis instances so that they don’t step on each other (using the Redis databases feature doesn’t quite work here). For a while we tried running these Redis instances on spot, but ugh, the monitoring noise and random breakage from Redis pods coming and going and then the app connecting to the read-only pod and throwing exceptions…. it was too much. The fix: get them off of spot instances. [this only affects beta/staging — those environments use a vendored Redis Helm chart that we run in the cluster, production uses Elasticache]We’ve successfully run lots of things that require PVCs in Kubernetes. Heck, for several months we ran the entire Elasticsearch clusters for Basecamp 2 and 3 on Kubernetes without any major issues. But that doesn’t mean I’d recommend it, and I especially do not recommend it when using spot instances. One recurring issue we saw was that a node with a pod using a PVC would get reclaimed and that pod would not have a node to launch on in the same AZ as the existing PVC. Kubernetes absolutely loves this shows this via a volume affinity error and it frequently required manual clean-up to get the pods launching again. Rolling the dice on having to intervene whenever a spot reclamation happens it not worth it to our team.



Getting it right



node-termination-handler

Arguably the most important piece of the spot-with-Kubernetes puzzle is aws-node-termination-handler. It runs as a DaemonSet and continuously watches the EC2 metadata service to see if the current node has been issued any spot termination notifications (or scheduled maintenance notifications). If one is found, it’ll (attempt to) gracefully remove the running pods so that Kubernetes can schedule them elsewhere with no user impact.

cluster-autoscaler

cluster-autoscaler doesn’t need many changes to run with a spot auto-scaling group. Since we’re talking about ensuring that instance termination behaviors are handled appropriately though, you should know about the pain we went through to get ASG rebalancing handled properly. When rebalancing, we’d see nodes that just died and the ALB would continue to send traffic to them via kube-proxy because there was no warning to kubelet that they were about to go away. aws-node-termination-handler cannot handle rebalancing currently, but I believe that EKS managed nodegroups do — at the expense of not supporting spot instances. A project called lifecycle-manager proved to be critical for us in handling rebalancing with ease (even though we ended up just disabling rebalancing all together

Black Lives Matter

Today, on Juneteenth, it is important to us to take a pause and publicly recognize organizations, activists, campaigns, and journalists that have been doing the work, fighting everyday for a more just tomorrow.

The following are contributions from individual Basecampers.

Jabari, Burlington, VT, US

Organizations:

Cooperation JacksonThe Okra ProjectThe Bail Project

Reads:

The Invention of the White RaceJackson Rising: The Struggle for Economic Democracy and Black Self-Determination in Jackson, MississippiFearless Cities: A guide to the global municipalist movement

Kristin, Portland, OR, US

Organizations:

Portland African American Leadership Forum Action Fund: https://paalfactionfund.org/defundpoliceDon’t Shoot Portland: https://www.dontshootpdx.org/

Readings:

End of Policing (Free Ebook y’all): https://www.versobooks.com/books/2426-the-end-of-policingFreedom is a Constant Struggle: https://bookshop.org/books/freedom-is-a-constant-struggle-ferguson-palestine-and-the-foundations-of-a-movement/9781608465644

If you live in Portland, please email Mayor Ted Wheeler MayorWheeler@portlandoregon.gov and City Commissioner JoAnn Hardesty JoAnn@portlandoregon.gov to tell them what you think of their new budget, which doesn’t defund the police in a meaningful way.

Joan, Portland, OR, from South Dakota, US

Organizations:

Equitable Giving CircleCheyenne River Youth Project

Reads:

So You Want To Talk About Race (available as a free audio book from Multnomah County Public Library, with no waiting. Check if your local library has some free resources.)

Lexi, Brooklyn, NY, US

Local organizations:

Warriors in the Garden: A collective of activists dedicated to peaceful protest for legislative change. Up to the minute protest scheduling and news via their Instagram. Black-Owned Brooklyn: Black-Owned Brooklyn makes it easier for you to discover local Black-owned businesses, in addition to documenting Black life, past and present, in the borough. We seek to lift up and preserve rich stories that are often overlooked in gentrified Brooklyn and encourage you to learn more about — and participate in — the creativity, beauty and self-determination throughout our community.

Follow: The Conscious Kid on Instagram: Parenting and Education through a Critical Race Lens.

Listen: The 1619 Project: An audio series on how slavery has transformed America, connecting past and present through the oldest form of storytelling.

Elizabeth, Longmont, CO, US

In Colorado, the Enhance Law Enforcement Integrity bill, SB20-217, will be signed into law today (6/19/20). Read up on the changes that it will bring.

Organizations:

Denver Justice Project works with historically marginalized communities to address systemic racism by transforming law enforcement and the structure of the criminal justice system through intersectional movement building, direct action, advocacy, and collaborative education. Volunteer interest formColorado Criminal Justice Reform Coalition: advancing a broader debate and design of public health, safety and funding strategies through collective action. Numerous initiatives and ways to be involved.IREE (Institute for Racial Equity and Excellence): Professional development, family advocacy, “to identify and eliminate practices and policies that privilege ‘some’ at the expense of ‘others’. To dismantle the preschool to prison pipeline.”Colorado People’s Alliance  and Colorado Climate Movement: Environmental and climate justice.

Wailin, Oak Park, IL, US

Freedom to Thrive is a community group working to reimagine community safety in our town. This last week’s action was a letter-writing campaign asking our two local school districts to terminate their contracts with the police department for School Resource Officers in the middle and high schools. For anyone interested in learning more about removing School Resource Officers and ending the school-to-prison pipeline, check out Dignity in Schools.“An Interactive Guide to Ambiguous Grammar” by Vijith Assar was published in McSweeney’s in 2015 but is just as relevant today.Speaking of being a critical reader of the news, Block Club Chicago and City Bureau are showing how coverage of a city can be done in an inclusive, community-based way. Another great organization is the Ida B. Wells Society for Investigative Reporting, which works to train and retain investigative reporters and editors of color. Nikole Hannah-Jones, who won a Pulitzer for the 1619 Project at the New York Times Magazine, is one of the co-founders.

Jane, Chicago, IL, US

The Chicago Alliance Against Racist and Political Repression, Black Lives Matter Chicago, and over 70 other organizations are working towards the passage of a Civilian Police Accountability Council (CPAC) ordinance. This ordinance would give communities far more oversight over the Chicago Police Department. If you live in Chicago, please call your alderperson to support CPAC, donate to the cause, and tell all of your neighbors.June is also Immigrant Heritage Month. Upwardly Global supports immigrants, refugees, and asylees in utilizing their full breadth of skills in the American workforce. Upwardly Global champions jobseekers in all industries and since COVID, they have doubled down on efforts in the healthcare space specifically, from helping doctors relicense to connecting jobseekers with contact tracing training.

Navid, Chicago, IL, US

Organizations

Chicago Freedom School provides leadership development and teaches community organizing skills to POC youth. While doing this, they also help mitigate barriers such as food insecurity and lack of internet access. During the first weekend of protests they took in protestors after Mayor Lori Lightfoot issued a curfew and raised the bridges, effectively trapping protestors downtown. The city retaliated by issuing a bogus citation.Brave Space Alliance “is the first Black-led, trans-led LGBTQ Center located on the South Side of Chicago, dedicated to creating and providing affirming, culturally competent, for-us by-us resources, programming, and services for LGBTQ individuals on the South and West sides of the city.”Invisible Institute. Definitely worth donating your money to. Because of their work in a 2014 lawsuit, the state government has ruled data on police misconduct as public information. They are continuing to do investigative work on human and civil rights abuses. They could use our support right now.

If you live in Chicago, you can find your Alderman’s info here. Tell them to vote for CPAC and defund the police.

Andrea, Chicago IL, US

Majostee Allstars: an incubator for the personal, cultural and social development of youth in Chicago’s most under-resourced communitiesThe Marshall Project: nonpartisan news organization that seeks to create and sustain a sense of national urgency about the U.S. criminal justice systemTeaching Tolerance: a bunch of free resources from the Southern Poverty Law Center for talking to kids about racism, anti-bias, and social justiceThe Dignity of Resistance by Roberta M Feldman & Susan Stall; a book that chronicles the four decade history of Chicago’s Wentworth Gardens public housing residents’ grassroots activism. Explores why and how women residents creatively and effectively engaged in organizing efforts to resist increasing government disinvestment in public housing and the threat of demolition.

Troy, San Antonio, TX, US

Organizations

RAICES is doing tireless work to protect the rights of immigrants and refugeesFix SAPD is focused on removing state statutes and contract policies that create barriers for accountability and change in San Antonio.

Reading and Listening

How to Be an Antiracist Highlights why “not racist” is not a thing. You are either working for equality or preserving the status quoThe New Jim Crow: Mass Incarceration in the Age of Colorblindness A devastating look at how the criminal justice system is designed to be so oppressive.Algorithms of Oppression: How Search Engines Reinforce Racism A reminder that technology is not neutral and racism and sexism are at the root of some of the basic tools we build and use.Scene on Radio podcast Season 2: Seeing White Probably the single most impactful thing I’ve gone through yet. Season 3: MEN and Season 4: The Land That Never Has Been Yet also look amazing.

Jim, Manchester, United Kingdom

The Trussell Trust and FareShare are two organizations working to mitigate the effects of poverty, by running food banks and ensuring that vulnerable people are able to eat. The impact of COVID-19 has increased the need for these services, especially for Black, Asian and minority ethnic (BAME) people and disabled people.The work of The Black Curriculum is both vital and interesting – the failure of the existing curriculum to implement the recommendations of the Macpherson Report is something that should be remedied. The recent report by the Runnymede Trust: Race and Racism in English Secondary Schools by Dr Remi Joseph-Salisbury covers these topics too, along with discussion on the presence of police in the school environment. I just started reading Natives by Akala, which digs into race and class in the ruins of empire.It’s Pride, and both Stonewall and Mermaids are working for LGBT rights in the UK, and supporting members of the community, often against a backdrop of fear and transphobia in the UK media.Next on my reading list after Akala is Hood Feminism, by Mikki Kendall. 

James, Berlin, Germany, originally from England

Organisations

Center for Intersectional Justice: aims to make anti-discrimination and equality policy more inclusive and address structural inequalities more effectively in EuropeISD-Bund: the Initiative Schwarze Menschen in Deutschland, which works to fight racism and lift up Black projects and voices, here in GermanyGive Something Back to Berlin: a platform and network which aims to bridge the gaps between migrants from different walks of life Refugio: a project of Berliner Stadtmission, where new and old Berliners live and work together

Listenings:

About Race: following the book Why I’m No Longer Talking (To White People) About Race, Reni Eddo-Lodge catches up on the state of race relations today, in the UKDissect examines a single album per season, one song per episode, currently focused on Beyonce’s LemonadeEar Hustle: the daily realities of life inside prison, produced within the walls of San QuentinHistory: The Shequel: self-proclaimed loudmouth Erin Gibson highlights the women that textbooks overlookThe Nod: an archive of “all the beautiful, complicated dimensions of Black life”, now a show on QuibiOrdinary Equality: a comprehensive history of the Equal Rights AmendmentPod Save The People: news, culture, social justice, and politics, from organiser and activist DeRay MckessonScene on Radio’s Seeing White series: a critical look at racism – usually represented as a Black issue – through the lens of whitenessWaiting on Reparations: a show about hip hop and politics, presented by rapper-activists Dope Knife and Linqua Franqa

Willow, Spokane, WA, US

Organizations

The Black Lens – Spokane’s local, independent, Black led newspaper.Spokane Community Against Racism (SCAR) – Did you know that Spokane has the 5th deadliest police force (by population) in America? Changing that is high on the list for SCAR, as is education equity, court monitoring, and more.WAISN – WA Immigrant and Solidarity Network works across the state with rapid response teams to respond to ICE attacks, they also work on deportation defense, court accompaniment, and have a bond fund that helps immigrates stay and fight deportation cases.

– Read –

Beyond Survival: Strategies and Stories from the Transformative Justice Movement How do we handle harm in our communities? What do people do when they don’t call the police? Timely questions, but not new, this collection of essays point at ways to answer how to re-imagine what justice and accountability look like.Deciding for Ourselves: The Promise of Direct Democracy How do communities self-organize? How can we keep the momentum of various groups working together for a common good going? These essays give us models for what other communities have experienced, and hope for how to envision and work toward more just communities.

Celebrating 3 million accounts (fewer)

We recently deleted over three million accounts across all our apps. This was the answer to a question we asked ourselves last year: what should we do about  accounts that weren’t cancelled, but weren’t used either? Should we keep hold of their data forever? 

That felt wrong – we promise to delete data when you cancel your account. Keeping so much data around felt like we weren’t living up to that promise, and felt like a liability, so we decided to do something about it.

We planned to target three groups of accounts, representing 3.2 million across six apps: Campfire, Backpack, Highrise, and the Basecamps (Classic, 2 and 3):

Paid accounts who stopped paymentsOld trial accounts that never upgraded to a paid accountFree accounts that haven’t been active in over a year

We organized the project in stages, each focused on a specific group of accounts. We implemented two actions for each:

A big initial deletion of accountsAn automatic cancellation workflow to keep the system self-maintained moving forward.

For the initial cancellation of accounts, we had a thorough discussion about whether we should notify them all. We had accounts dating back to 2004, and with the number of accounts we were talking about, even a small percentage of replies would represent a significant extra workload for our support team. Also we worried about former customers thinking we were spamming them. How could we do the right thing, in a manageable way? We decided on some thresholds to try and find the right balance. For example, we decided to cancel without notifying accounts that had been inaccessible for not paying for three years or more.

Then a customer wrote in about his old account (from 2004!). He had stopped paying in 2015 and hadn’t accessed their account since then. And now, almost 5 years later, they wanted to reopen it. When they wrote us, they were beyond the 30-days limit we wait between cancellation and actual data incineration, so despite trying hard, we couldn’t recover their data. They were gracious and understanding, but it was heartbreaking. We decided to change course: we would notify everyone instead. We talked about this in detail in this episode of Rework.

Sending millions of emails made us work hard on making them as self-serving as possible, for our customers, and for our support team. We refined, measured, and refined again in a process that saw us deliver 200 emails per day in December to 28,000 per day when we finished in April. For example, these are the initial and final versions we used for notifying free inactive accounts.

[image error][image error]

This is the chart for the backlog of free inactive accounts (the glitch in the middle was due to a bug that meant a bunch of emails weren’t sent, so we had to resend):

[image error]

This project involved many people including Jane Yang leading it, Justin White taking care of the first half of the project and me completing it. We had to review our apps to measure account activity, implement a cancellation workflow that adapted to each product, and prepare our data-deletion queues to work with these kinds of numbers. Our support team handled over 2000 tickets related to this. As a new-hire, the amount of effort Basecamp was willing to put into doing the right thing was impressive. After all, this was not a new feature or product, but the kind of work that often goes unnoticed.

This was an example of the many things we are doing to raise the bar on data privacy. We will share more. Stay tuned!

On current events

It’s easy to say what a year, what a week.

But that’s a shortsighted, privileged point of view. I’m guilty of holding that occasional perspective. It’s moments like these that jolt me into recognizing the deeper reality.

What we’re seeing is the culmination of years – decades, generations, and centuries – of unjust treatment against black people, minorities, and other marginalized communities.

This country’s racist history is shameful, and so is its present.

Deep systemic racism + the militarization of police (both physically in terms of gear, and mentally in terms of mindset) is a powder keg. We’ve seen sparks before, now we’re seeing the explosion.

If you’re surprised, you’re not paying attention.

I don’t like the violence, but I get it. This is what happens when people are squeezed, compressed, and backed into a corner with no way out. For years, for generations. We’re all humans – if your lot in life was different you just might do the same.

I support peaceful protests, I support the fight against racism, against oppression, and against injustice – wherever it hides.

There’s exceptionally hard work ahead. I recognize this work has been happening for years, often ignored or unappreciated by many people, including me. How frustrating it must be to work so hard, and see such little progress, on something so elemental.

Change will require a massive, sustained effort by millions over many years. A change in perspective, mindset, and approach. And that work will certainly be met with future setbacks, which is why change requires optimism, too (which is in short supply in moments like these). I hope we can find it, and support those who need it.

I’ll be working to educate myself, and break my own patterns of ignorance. This sense of urgency is, embarrassingly, new to me, so I have a lot to learn – which organizations to support, what books to read, what history to absorb, and who to listen to. I’m starting on that today. If you’re like me, I hope you’ll do the same.

-Jason Fried, CEO, Basecamp

Employee-surveillance software is not welcome to integrate with Basecamp

We’ve been teaching people how to do remote work well for the better part of two decades. We wrote a whole book about the topic in 2013, called REMOTE: Office Not Required. Basecamp has been a remote company since day one, and our software is sold as an all-in-one toolkit for remote work. Yeah, we’re big on remote work!

So now that COVID-19 has forced a lot of companies to move to remote work, it’s doubly important that we do our part to help those new to the practice settle in. We’ve been hosting a variety of online seminars, done podcasts, and been advocating for healthy ways to do remote right.

Unfortunately, the move to remote work has also turbo-charged interest in employee surveillance software. Drew Harwell’s harrowing report for The Washington Post should make anyone’s skin crawl, but it seems some managers are reading about these disgusting tools and thinking “oh, what a great idea, where can I buy?”.

And as faith would have it, some of those managers would then visit these employee surveillance vendors and see a Basecamp logo!