Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley

by Antonio García Martínez

FBX wasn’t merely a new ads feature. It was an entirely new ads system parallel to the existing Facebook one. The exchange would have separate data stores with billions of lines of data, reflecting the various data joins to the outside world we had made. The FBX code itself, unlike anything inside Facebook Ads, would have to contact machines all over the world for bids and ads in real time, and react to bids just as quickly, all of this tens of billions of times per day, hundreds of thousands of times per second. How ads themselves were created and uploaded would be different, allowing advertisers that much-coveted (and before FBX impossible) ability to dynamically fine-tune ads for every user and every ad experience. Ads statistics, the boring bean-counting task critical to keep the budgets flowing, would have to exist separately and work for advertisers and their accounting departments. Not to mention a whole suite of internal tools and dashboards to monitor the health of this beast we had created, making sure bids weren’t getting dropped, or we weren’t slamming our partners with too many requests for bids. On the business side, we (by which I mean the royal PM “we”) would have to find advertising software companies as partners, and on-board them with a list of documents, specifications, and integrations tests.