Site Reliability Engineering: How Google Runs Production Systems

by Betsy Beyer

In the spirit of nurturing the postmortem culture, SREs proactively create activities that disseminate what we learn about system infrastructure. Some example activities include: Postmortem of the month In a monthly newsletter, an interesting and well-written postmortem is shared with the entire organization. Google+ postmortem group This group shares and discusses internal and external postmortems, best practices, and commentary about postmortems. Postmortem reading clubs Teams host regular postmortem reading clubs, in which an interesting or impactful postmortem is brought to the table (along with some tasty refreshments) for an open dialogue with participants, nonparticipants, and new Googlers about what happened, what lessons the incident imparted, and the aftermath of the incident. Often, the postmortem being reviewed is months or years old! Wheel of Misfortune New SREs are often treated to the Wheel of Misfortune exercise (see “Disaster Role Playing”), in which a previous postmortem is reenacted with a cast of engineers playing roles as laid out in the postmortem. The original incident commander attends to help make the experience as “real” as possible.