Androids: The Team That Built the Android Operating System

by Chet Haase

“Any time anything works out, if you don’t acknowledge the huge luck factor, you’re kind of a jerk.” –ficus kirkpatrick

“We didn’t know if it was going to totally flop or if it was going to work. When it worked, I think people were as surprised as they were excited.” –Evan Millar

“That’s part of this world: you learn probably more from the failures than you do from the successes.”

“When Andy said, ‘The company’s gonna be bought by Google,’ I thought ‘Wow, this is the only way I could ever possibly get into Google.’ Then he said, ‘We have to go in and interview,’ and I thought, ‘Well, that’s it. It’s over.’”

Swetland remembered, “Ficus declared that if anyone asked what the Big O51 of anything was, he was simply going to say, ‘I’m far too handsome to answer that question.’”

By having a single platform that worked across all devices using it, Android would simplify phones for both manufacturers and developers.

“They bought a team and a dream. I’d like to believe we executed pretty well on it.” –Brian Swetland

it was difficult to build Android applications without having the underlying OS kernel, graphics system, framework, UI toolkit, APIs, and other fundamental layers that those applications would need. Because there’s nothing worse than walking into your new penthouse apartment and falling down all the way to street level.

“Everything I was building was toward something that would eventually be a product. I’m not a believer in doing total one-off demos. We didn’t have process separation,70 but we knew where that was going to go. We still needed the

kernel, the bootloader,71 the graphics drivers, and everything. We did some demo-y things along the way, but we always tried to chart a path where it wasn’t pure demo. It was progress towards a system.”

“The dangerous thing of building purely demo-ware is someone decides you’re going to ship it. Then you’re screwed.”

So Arve added the concept of wake locks to Android’s Linux kernel, to ensure that screen-off didn’t mean completely-off. Android would aggressively put applications and most of the system to sleep when the screen turned off (because battery consumption was always a huge concern), but wake locks ensured that the system could stay in a wakeful state if there was something happening that needed to continue even if the screen turned off.

“Someone had decided that having the SD card on the side, hot-swappable, was a good idea. If you try to pull a hard drive in and out of a Linux system, you’re gonna have a bad time. This was the worst: device ripped out with almost no warning. You could be writing to the disk. Maybe you took a picture thirty seconds ago, so those buffers were still in the OS page cache, and they’re not gonna get written out for another thirty seconds.”

There was a cover on the G1’s SD Card slot that users had to open before popping the card out. Opening that cover sent a signal into the system, which could be used as a cue to quickly get everything into a stable state in case the user popped the card out. But it was difficult to find all of the places in the code where this needed to happen. To make things worse, debugging the situation required a lot of tedious card-popping. Over, and over, and over. Eventually, San asked for help. He reached out to Andy: “‘Hey, you’re into robots. Do we have someone that could make me a robot that could do this?’ He hooked me up with someone. I told them what I needed: a little bot that allows me, through software, to control how it goes in and out, and then I would create a little closed-loop test. That allowed me to track down all of these bugs.” San used the SD card robot to chase those bugs one by one until the system worked reliably.

“We built this safe. We just forgot the walls.” –Rebecca Zavin

“We got this weird bug from someone saying they’d SSH into their server at home, type ‘reboot’, and their phone would reboot. We thought they must be smoking something, and closed as Not Reproducible.”92

“At some point, we had this ongoing bug where we used to have the framebuffer console support there, so you could switch to a mode where you could see the log, same as you can on your Linux PC. We had this recurring bug where you’d get a black square in the upper left corner. It was a timer issue. Some race condition96 would cause the cursor to blink when you came back to graphics. So I’d get Steve Horowitz [telling me], ‘That black square. I got a black square!’ “After spending a lot of time trying to fix the problem, I was like, ‘Let’s just turn off the framebuffer console. Why do we need to see the kernel log on the screen on the device? This is stupid. Let’s just turn it off and then we don’t have to deal with this problem.’ “But when we turned off the console, it was still there [just invisible]. We all were like: ‘Uh-oh, oops!’ “We built this safe. We just forgot the walls.” Jeff said, “The person typing ‘reboot’ into ConnectBot was effectively typing it into both their remote server, and into their phone, which explains the surprising reboot.” Nick Pelly remembered the bug: “We thought it was some subtle, clever hack that someone had figured out. No, no, no: every single keyboard press you made was going into root shell.”

“We felt we were going to change the world. And we did.” –Mike Chan

But the Android engineers felt (rightly so) that the applications on the device should not be inherently trusted. So rather than having apps running as the user who installed them, Brian’s design had each app run as a separate, unique user on the device. This approach guaranteed (through the Linux kernel mechanism of user IDs (UIDs)) that apps had no automatic access to the data of any other application on the same device, even though those other applications were installed by the same device owner. Brian provided a low level service to create, destroy, or run-as-a-user. Dianne Hackborn, on the framework team, integrated this service with higher-level application permissions and built out the policies for application UID management.

“Steve Horowitz [Android’s engineering director at the time] was all over me. ‘You broke the build!’ I’m like, ‘I know, Steve. I know I broke the build. I’m trying to fix it now. Having you stand there is not going to make me fix it any faster.’

Mike quickly realized that part of the problem was one of expectations. “The problem I explained to everyone was: You’re telling me we have to have just as good battery life as the iPhone. We have this capability to run all these apps in the background,99 the hardware that we have has a bigger screen, we run background tasks, we were the first to do 3G, and we have a physically smaller battery.”

Mike also had ongoing debates with Dianne, on the framework team. Many of the battery problems were coming from poor behavior in apps, which would do things like hold a wake lock100 too long, but the users would just blame Android overall. “I was pushing for a more explicit system where if an app goes into the background, you force-release the resource. So basically a less flexible platform. Dianne was under the firm belief that it’s not the platform’s fault, it’s the developers’, and the right fix is to educate all these app developers. “This was a battle we had for years.”

A governor in an OS is the mechanism by which the speed, or frequency, of the CPU is changed to save power. For example, if your CPU is running really fast, then it is consuming more power and thus more battery. But if the device is idle at the time, that’s a large and unnecessary waste of battery power. The governor exists to detect these different runtime modes and scale the CPU frequency accordingly. When the G1 launched, the only governor in effect was the ondemand governor that was part of core Linux. It was a simple system with just two settings: full-speed and idle. This was better than nothing, but wasn’t good enough for Android’s purposes, especially since the heuristics for that governor were tuned for Linux running on server or desktop machines, not for the more constrained world of mobile devices. Mike started playing with the governor in the late stages of 1.0, but had to put that project aside after an unfortunate incident of a demo that Andy gave to the Google execs. Mike had checked in a change that inadvertently made the phone painfully slow. “I was experimenting on master101 with the conservative governor, which skewed very, very heavy towards saving power at the cost of performance. The phone was pretty much unusable.” Meanwhile, Andy had a monthly review with Larry and Sergey to show them the status of this project. Andy flashed102 a build of master onto a phone and went off to his meeting. At the meeting, he showed a demo using that build, which… did n...

“Our original idea was that we were going to do everything in a language-independent way. You were going to be able to write your apps in JavaScript, C++, and Java. Eventually we realized there were like twelve of us and we were never going to make that work. So we said ‘OK, we have to pick a language.’”

Dan began work on a new runtime, which he named Dalvik: “I had just finished reading an issue [of McSweeney’s], which consisted of English translations of modern Icelandic fiction. So I had Iceland on the brain. I looked at a map of Iceland and tried to find something that was short and pronounceable, and didn’t have any of the weird characters, and I found Dalvík114 (or ‘Dal-veek,’ as it’s pronounced). It sounded like a nice little town.”

One of the things that the Dalvik team created to make Android work for 1.0 was (and still is) called Zygote.117 Zygote is like the loaf of bread you slice when making a sandwich. You could, of course, bake the bread from scratch every time you made a sandwich, but that’s a lot of effort and time spent every time you want a sandwich. It’s obviously much faster and easier to have a loaf that you can just cut slices from to jump-start each sandwich. Zygote is like sandwich bread for applications.

Dan had this idea, which came from a feature of Emacs (a popular text editor on Unix systems), where it would allow you to dump the state at any time and then start up Emacs later from this saved state (cleverly called undump). This meant that Emacs could start much faster because it just sucked in the state from disk instead of running a bunch of code logic at startup time. “My idea was that we implement an undumper-type system as made most ‘famous’ (at least to me) by Emacs. Mike

[Fleming] said, ‘How about we skip the part where we dump to disk and reload?’ And he ran with it.” Mike got the system up and running, dramatically changing the way that applications started. Instead of each app loading in all of the code they required, and initializing it as it was loaded in, the Zygote system created a single process with much of the core platform code, essentially pre-loading and initializing all of it. Whenever an application launched, the Zygote process would be for...

Bob Lee (who worked on core libraries, the subject of the next chapter) said of Zygote, “It was just so simple! It’s like one API call! The reason we were able to do that was that the memory was copy-on-write.118 So as long as you didn’t touch those memory pages from that initial Zygote process, all that memory would be shared across the whole OS. It was just such a clever, beautiful solution to leverage stuff that was already there.” The system didn’t quite work as intended at first. Bob chased down a problem with the garbage collector: “After one garbage collection, I was like, ‘My app’s taking up so much memory again!’ It was because the garbage collector would touch every memory page.” That is, the normal efforts of the runtime would write to pages in memory that needed to stay read-only for the shared-memory approach of Zygote to work. Fadden came in with a fix for this. Each new process would separate the heap from the garbage collector after the Zygote stage, excluding it from the memory that the garbage collector examined. The shared memory portion didn’t even exist in the new app, so it wouldn’t get touched. After this, Bob and Fadden continued to work on Zygote, to fig...

take up too much memory for the shared processes that’s only ever going to be used by one app.” Bob credits Zygote for Android being at all functional at that time: “The Zygote thing helped a lot, just being able to share memory, going from having just a couple Java processes running to having dozens running on a really small device. And rather than having to wait for a whole VM to start up, our apps actually looked faster; they would launch instantly, because we’d just fork a process and start right there. Everything was already warmed up.” Eventually, Zygote ...

As part of making the core libraries work, Bob implemented significant networking functionality, fixing bugs along the way. One of those problems prevented every phone from starting up at all. “The first time you started a phone, it had to connect to a time server, but the time [on the device] was set to sometime in 2004.” The phone would try to connect to the server through a secure connection, which

requires a security certificate on the server. But the initial time on the phone was before the time that the certificate was issued on the server, so the connection would fail and the phone wouldn’t boot. Bob’s fix was to catch that failure condition and set the initial time on the phone to the day that he fixed the bug. Bob also tracked down a networking problem that was specific to mobile data. Android phones were experiencing severe outages that seemed like a problem with bad carrier network infrastructure.

“We spent a long time taking these APIs and re-implementing them from scratch to be good, while maintaining their existing terrible APIs.”

‘Hey, heads up, the open source library for the JSON library we’re using has this, ‘The Software shall be used for Good, not Evil’ clause126 in it. That means it’s not open source because open source has no discrimination against any endeavor.’ So

“The first monkey lab was my laptop and seven Dream devices. I wrote some scripts and tools to beat the shit out of them till they crashed.”

“My idea was: I want everything to be ready for when we get hardware. The problem is, we don’t have hardware. We don’t really know when it’s going to happen. So I thought, I’m in charge of graphics, I’m going to pretend I have a GPU. So I wrote a GPU, essentially. This way, I was able to write SurfaceFlinger using ‘GL.’ It was using real OpenGL ES, but it defaulted to software. And then, little by little, real hardware started to show up.” When Mathias said that he wrote a GPU, he meant that he wrote a virtual GPU; software that performed the same work as a GPU would, but in software instead of dedicated hardware. A GPU is not magical; the dedicated hardware in a GPU is not doing anything that cannot be done by software running on the CPU instead. It just does that job much faster, since it has hardware that is optimized for graphics operations.150 In writing his fake GPU, Mathias provided a software layer to handle graphics operations normally handled by a GPU, translating those commands to low-level information that the existing Android display system could understand.

Once PixelFlinger and OpenGL ES were working, Mathias could implement SurfaceFlinger. Applications draw their graphics objects (buttons, text, images — whatever) into a buffer in memory and SurfaceFlinger posts that buffer to the screen, visible to the user. SurfaceFlinger was essentially the glue between the higher-level graphics operations happening in applications and the OpenGL ES layer that he had written previously, responsible for copying buffers around and displaying them to the user. The separation of app rendering from displaying the pixels on the screen was intentional; one of Mathias’s design goals was to enable smooth graphics by ensuring that no app could cause rendering performance problems for any other app (this was related to Android’s overall approach to security on the platform, where there was always a clear separation of applications from each other). So applications would draw into buffers, and SurfaceFlinger would take it from there.

The other part of the graphics system that Mathias wrote was the Hardware Composer (HWC). SurfaceFlinger handles drawing UI graphics into a window on the screen. But there are several windows that need to be combined together to make up the final pixels on the screen. Think about a typical Android screen that the user sees. There is a status bar (where the current time and various status and notification icons show up), a navigation bar (where the back and home buttons reside), and finally the actual foreground application (or the home screen). There may be other windows as well, like popup menus on top of the foreground application. A sample Android screen, showing the typical status bar, navigation bar, and home screen. All of these are separate windows, often running in separate processes. For example, the navigation and status bars are managed by the system process, whereas the application window is owned by the application process. All of these windows need to be displayed together in some sensible way, which is the job of the Hardware Composer. Mathias’s idea for HWC was to use specialized graphics hardware called hardware overlays,155 which provide display memory dedicated to each application, avoiding the overhead of all applications sharing the same video memory. Using overlay hardware also saved power and provided higher performance for applications. By using dedicated overlay hardware, the system avoided using the power-hungry GPU for these simple and frequent w...

Dave remembered the deal as, “This is going to be open sourced in the future. Here’s a pile of money.”

Similarly, AudioFlinger combined multiple audio streams across the system into a single stream which would then be sent to the driver without (and this is the key part) causing the device to reboot.

Whenever Dan Sandler visited the Mountain View offices, he’d usually leave behind some whiteboard art in his wake. This one showed up in the framework area on one of those trips.

“And it’s chicken and egg. You have to get users to get app developers interested in you, and you have to get developers to get users interested in you. You can get some users, but anywhere you try to gain momentum, the dominant platform can go address that particular market and kill you. It’s just impossible.”

The hardware was getting more powerful, and the market was already bigger than PCs.”

“It was really hard to get phone manufacturers interested in someone else’s platform. They did their own software and they were terrified of phones becoming the same as PCs, where there’s one software vendor that owns the platform that commoditizes the hardware.”

“It’s hard to out-code Dianne when she’s on fire. She had a vision of what she wanted, and she just sat down and typed it out.” –Jeff

The 3D launcher was a special effect in the all apps screen that lasted for several releases. The user saw a normal 2D grid of applications, but as they scrolled the list up and down, the top and bottom edges faded away into the distance like a Star Wars intro-text effect. It was subtle but powerful, hinting at the 3D power behind the system (and the potentially large number of apps on the system), but without being too ostentatious or difficult to navigate.

“It was almost universally acknowledged that Android nailed the notification system on day one. It would take iOS another three years before launching a design as effective at triaging

messages and alerts coming from users’ ever-growing collection of mobile apps. The secret was in the G1’s unique status bar, which could be dragged downward to reveal every notification in a single list: text messages, voicemails, alarms, and so on. The fundamental concept lives on (in a refined form) to today’s latest version of Android.”

Design is everything. Design is how people see a product, what they feel when they use a product. It’s part of why something succeeds and something fails. –Irina Blok

“When we were first starting, Larry and Sergey, probably Sergey, insisted on no animations,239 because they were a waste of time. Which to look at phones now… That’s kind of why Android was a little more spartan.”