Healthchecks at scale.

A couple of days ago at Stripe's weekly incident review, we started a discussion on a topic that is always surprisingly controversial: healthchecks. I've been thinking about them since and have written up what I've learned about healthchecks.

When I joined Uber, we relied on HAProxy to route traffic between processes. HAProxy was configured in what might now be described as the sidecar pattern.

We ran one instance of HAProxy on every server, and generated a server's routing configuration by inspecting metadata stored...