Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2

by Thomas A. Limoncelli

Imagine How Behavior Will Change

What are all the ways that people could behave but still match the incentive? How could a person maximize his or her personal gain?

engineers will examine the KPIs given and follow them as stated. Again, this is not “gaming the system,” but simply following the rules.

Revise and Select

we revise the KPI. We may select one KPI over another or loop back to Step 1 and start over.

Seek confirming and non-confirming data.

Deploy the KPI

or institute it as policy. This is mostly a communication function.

Deploying the KPI means making people aware of it as well as putting into place mechanisms to measure the KPI.

the process should not require human intervention.

Step 5 deploys the KPI. The KPI is communicated to key stakeholders. Measurements required to calculate the KPI plus the computation itself should be automated and presented in a dashboard.

Evaluating the KPI A few weeks after deployment, the initial results should be audited for unintended negative side effects.

Error Budget Benjamin Treynor Sloss, Vice President of Engineering at Google, revealed a highly successful KPI called Google Error Budget.

When the budget is exhausted, all launches stop. An exception is made for high-priority security fixes.

Remember that the KPI should be achievable.

High-performing organizations have smooth operations, well-designed policies and practices, and discipline in what they do. They meet or exceed the needs of their customers and delight them with innovations that meet future needs often before such needs ever surface. The organization is transparent about how it plans, operates, provides services, and handles costs or charge-backs to customers. The vast majority of customers are happy customers. Even dissatisfied customers feel they have a voice, are heard, and have a channel to escalate their issues. Everyone feels the operations organization moves the company forward. Its funding reflects a reasonable budget for the work the organization does. The organization makes its successes visible and, more importantly, is honest and forthright when it comes to discussing its own faults. The organization is constantly improving. Outages and escalated issues result in action plans that reduce future occurrences of that problem. The world is constantly changing, and the organization incorporates new technologies and techniques to improve its inner workings as well as the services it provides.

Operational Responsibilities

Regular Tasks (RT):

Emergency Response (ER):

Monitoring and Metrics (MM):

Capacity Planning (CP):

Change Management (CM):

New Product Introduction and Removal (NPI/NPR):

Service Deploy and Decommission (SDD):

Performance and Efficiency (PE):

look-for’s

A look-for is a behavior, indicator, or outcome common to a service or organization at a particular level.

A.1 Regular Tasks (RT) Regular Tasks include how normal, non-emergency, operational duties are handled—that is, how work is received, queued, distributed, processed, and verified, plus how periodic tasks are scheduled and performed. All services have some kind of normal, scheduled or unscheduled work that needs to be done. Often web operations teams do not perform direct customer support but there are interteam requests, requests from stakeholders, and escalations from direct customer support teams. These topics are covered in Chapters 12 and 14. Sample Assessment Questions • What are the common and periodic operational tasks and duties? • Is there a playbook for common operational duties? • What is the SLA for regular requests? • How is the need for new playbook entries identified? Who may write new entries? Edit existing ones? • How are requests from users received and tracked? • Is there a playbook for common user requests? • How often are user requests not covered by the playbook? • How do users engage us for support? (online and physical locations) • How do users know how to engage us for support? • How do users know what is supported and what isn’t? • How do we respond to requests for support of the unsupported? • What are the limits of regular support (hours of operation, remote or on-site)? How do users know these limits? • Are different size categories handled differently? How is size determined? • If there is a corporate standard practice for this OR, what is it ...

http://www.youtube.com/watch?v=Fx8OBeNmaWw