Andrew Huang's Blog, page 20

Obituary: Gavin Zhao

If you look inside “The Essential Guide to Electronics in Shenzhen”, you will find the following inscription:

To Gavin Zhao

For opening my eyes to the real China.

You have been a great teacher and mentor;

I can do now what I once thought was impossible.

I hope you win your battle with cancer,

so that you can continue to mentor and inspire more people like me.

That was written about four years ago. Today, August 9, 2019 at 1:34PM, he lost his battle with cancer. He died while I was on an airplane flying from Singapore to China to see him, perhaps for one last time…seems I was a few hours too late.

As a professional, Gavin interacted primarily with me as a project manager. He was instrumental in helping to build Novena, Chibitronics, Fernvale, and many more projects big and small. What made him special was not that he was a genius in electronics or process engineering. His degree was in Western Philosophy: he understood how people worked, both in terms of their minds and their hearts. He thought deeply on all issues, big and small; formed his own opinions about government and politics, and as such, always had to straddle a fuzzy gray line living in China.

Part engineer, part troublemaker – we got along well.

I often referred to Gavin as my cultural Rosetta Stone. We used to spend long afternoons discussing politics in China, comparing the merits of democracy and communism. There are plusses and minuses to both philosophies. He would archive and share with me stories and posts censored by the Chinese government; I would bring him copies of the New York Times and new books to read. He could explain the deep meaning behind some subtle government actions that would almost seem routine to a Westerner. The problem is, coming from my American background, there are so many mind-blowing things to learn about Chinese politics, we could never have enough time together to discuss. We’d meet for tea at noon and before we knew it, the sun was down. I started the Betrusted project in part because I wanted to be able to spend more time learning from Gavin – unfortunately, it just wasn’t safe for us to correspond via the Internet about some of the ideas and concepts I wanted to learn, so our political discussions were always face to face. Betrusted will come too late for Gavin, but hopefully not too late for others.

Gavin studied many religions, including Buddhism, Christianity, Judaism, Islam and Confucianism. Of all the religions, Gavin felt Confucianism was his favorite. It is a very practical religion, built around the fabric of human society, and not around some abstract ideals. Because human fabric is messy, understanding how to apply Confucianism correctly essentially requires the study of philosophy: one cannot boil Confucianism into a series of “thou shalt/thou shalt not”s. So, as a practitioner of Confucianism, Gavin was always a very practical person, and always had a very positive attitude, even in his darkest times. He once pointed me to this passage: “子曰：「女奚不曰，其為人也，發憤忘食，樂以忘憂，不知老之將至云爾」”. It doesn’t translate well to English, but from his explanations, I felt the passage really reflected his character. Last month, while he was literally doubled over in pain, vomiting from the complications of his cancer treatment, I was holding his hand when suddenly he had a moment of lucidity. He looked up at my face, smiled at me through my tears, and proclaimed, “I am just a common man, why am I chosen to have the strength to be able to endure this pain?” Even in his darkest times, he was able to crack a joke.

Mortality is a subject that has weighed heavily on my mind. One thing I have decided is that it’s better to celebrate the living than to mourn the dead. Thus, while I am sad that Gavin has passed, I prefer to celebrate his life, and to focus my emotional energy on supporting his wife Lisa and daughter Coco who succeed him. There was a precious couple of years while his cancer was in remission, and I’m happy we celebrated the time that he had – during this time, he became an MIT Media Lab Director’s Fellow, and we visited Boston together. He touched the lives of many students. I remember he was so excited to visit MIT’s library and explore the section on Kantian philosophy. He helped on NeTV2, and we started on Betrusted together. We went to Tokyo and wandered the grounds of the Imperial Palace, where we found an old, grand tree standing among ruins. He declared that he always admired trees, and he could sit there and watch trees for hours. He wondered – “If we could talk to trees, what stories could they tell us?” So we sat together under a tree for an hour, and watched as its boughs waved in the wind, watched its leaves fall, watched as birds hopped among its branches. It was a true luxury to spend an hour doing nothing but watch a tree together, with a friend who had so little time. At the end, I relayed to him an anecdote I once heard about trees: “don’t feel bad for trees because they can’t walk; feel bad for humans because they have to.” Although Gavin will never walk by my side again, his memory will live on in my soul like that tree – grand, growing, enduring; nourishing in lean times; yet soothing to sit under on difficult, hot days. It will be a landmark that guides me through my remaining life. I celebrate that I had the privilege of being touched by such a good friend and teacher.

Gavin, by carrying your memory in my heart, I celebrate your life as I continue with mine. You may be gone from this world, but you still shape ours in many ways.

Thank you, Gavin. May you rest in peace.

Gavin and I in front of one of his favorite trees at the Imperial Palace in Tokyo, Japan

Name that Ware, July 2019

The ware for July 2019 is shown below.

ISA, I say! Back in the day when all it took was a couple 7400 series chips to talk to a computer…now we use a small computer just to properly negotiate power before talking for real over standards like USB-C.

Thanks to Nava for contributing this ware, found at Akihabalast, “Akihabara final waste disposal site!”.

Winner, Name that Ware June 2019

The ware for June 2019 is a Mitsubishi Volume Pro 500. Gratz to Jean for guessing it, email me for your prize! It’s a neat if not obscure bit of computing history.

Open Source Could Be a Casualty of the Trade War

When I heard that ARM was to stop doing business with Huawei, I was a little bit puzzled as to how that worked: ARM is a British company owned by a Japanese conglomerate; how was the US able to extend its influence beyond its citizens and borders? A BBC report indicated that ARM had concerns over its US origin technologies. I discussed this topic with a friend of mine who works for a different non-US company that has also been asked to comply with the ban. He told me that apparently the US government has been sending cease and desist letters to some foreign companies that derive more than 25% of their revenue from US sources, threatening to hold their market access hostage in order to coerce them from doing business with Huawei.

Thus, America has been able to draw a ring around Huawei much larger than its immediate civilian influence; even international suppliers and non-citizens of the US are unable to do business with Huawei. I found the intent, scale, and level of aggression demonstrated by the US in acting against Huawei to be stunning: it’s no longer a skirmish or hard-ball diplomacy. We are in a trade war.

I was originally under the impression that the power to pull this off was a result of Trump’s Executive Order 13873 (EO13873), “Securing the Information and Communications Technology and Services Supply Chain”. I was wrong. Amazingly, this was nothing more than a simple administrative ruling by the Bureau of Industry and Security through powers granted via the “EAR” (Export Administration Regulation 15 CFR, subchapter C, parts 730-774), along with a sometimes surprisingly broad definition of what qualifies as export-controlled US technology. The administrative ruling cites Huawei’s indictment for willfully selling equipment to Iran as justification for commuting a broad technology export ban upon Huawei’s global operations.

Going Nuclear: Executive Order 13873

If a simple administrative ruling can inflict such widespread damage, what sorts of consequences does EO13873 hold? I decided to look up the text and read it.

EO13873 states there is a “national emergency” because “foreign adversaries” pose an “unusual and extraordinary threat to national security” because they are “increasingly creating and exploiting vulnerabilities in information and communications technology services”. Significantly, infocomm technology is broadly defined to include hardware and software, as well as on-line services.

It’s up to the whims of the administration to figure out who or what meets that criteria for a “foreign adversary”. While no entities have yet been designated as a foreign adversary, it is broadly expected that Huawei will be on that list.

According to the text of EO13873, being named a foreign adversary means one has engaged in a long-term pattern or serious instances of conduct significantly adverse to the national security of the US. In the case of Huawei, there has been remarkably little hard evidence of this. The published claims of backdoors or violations found in Huawei equipment are pretty run-of-the-mill; they could be just diagnostic or administrative tools that were mistakenly left into a production build. If this is the standard of evidence required to designate a foreign adversary, then most equipment vendors are guilty and at risk of being designated an adversary. For example, glaring flaws in Samsung SmartTVs enabled the CIA’s WeepingAngel malware to listen in on your conversations, yet Samsung is probably safe from this list.

If Huawei has truly engaged in a long-term pattern of conduct significantly adverse to national security, surely, some independent security research would have already found and published a paper on this. Given the level of fame and notoriety such a researcher would gain for finding the “smoking gun”, I can’t imagine the relative lack of high-profile disclosures is for a lack of effort or motivation. Hundreds of CVEs (Common Vulnerabilities and Exposures) have been filed against Huawei, yet none have been cited as national security threats. Meanwhile, even the NSA agrees that the Intel Management Engine is a threat, and has requested a special setting in Intel CPUs to disable it for their own secure computing platforms.

If Huawei were to be added to this list, it would set a significantly lower bar for evidence compared to the actions against similarly classified adversaries such as Iran or North Korea. Lowering the bar means other countries can justify taking equivalent action against the US or its allies with similarly scant evidence. This greatly amplifies the risk of this trade war spiraling even further out of control.

Supply Chains are an Effective but Indiscriminate Weapon

How big a deal is this compared to say, a military action where bombs are being dropped on real property? Here’s some comparisons I dug up to get a sense of scale for what’s going on. Huawei did $105 billion revenue in 2018 – 30% more than Intel, and comparable to the GDP of Ukraine – so Huawei is an economically significant target.

Above: Huawei 2018 revenue in comparison to other companies or country’s GDP.

Now, let’s compare this to the potential economic damage of a bomb being dropped on a factory: let’s say an oil refinery. One report indicated that the largest oil refinery explosion since 1974 caused around $1.8 billion in economic damage. So carving Huawei out of the global supply chain with an army of bureaucrats is better bang for the buck than sending in an actual army with guns, if the goal is to inflict economic damage.

Above: A section of “The 100 Largest Losses, 1974-2013: Large Property Damage Losses in the Hydrocarbon Industry, 23rd Edition”.

The problem is, unlike previous wars fought in distant territories, the splash damage of a trade war is not limited to a geographic region. The abrupt loss of Huawei as a customer will represent billions of dollars in losses for a large number of US component suppliers, resulting in collateral damage to US citizens and companies. Even though only a couple weeks have passed, I have first-hand awareness of one US-based supplier of components to Huawei who has gone from talks about acquisition/IPO to talks about bankruptcy and laying off hundreds of well-paid American staff; doubtless there will be more stories like this.

Reality Check: Supply Chains are Not Guided Missiles

The EAR was implemented 40 years ago, during the previous Cold War, as part of an effort to weaponize the US dollar. The US dollar’s power comes in part from the fact that most crude oil is traded for US dollars – countries like Saudi Arabia won’t accept any other currency in payment for its oil. Therefore sanctioned countries must acquire US dollars on the black market at highly unfavorable rates, resulting in a heavy economic toll on the sanctioned country. However, it’s worth taking a moment to note some very important differences between previous sanctions which used the US dollar as a weapon, and the notional use of the electronics supply chain as a weapon.

The most significant difference is that the US truly has an axiomatic monopoly on the supply of US dollars. Nobody can make a genuine US dollar, aside from the US – by definition. However, there is no such essential link between a geopolitical region and technology. Currently, US brands sell some of the best and most competitively priced technology, but also little of it is manufactured within the US. US may have one of the largest markets, but it does not own the supply chain.

It’s no secret that the US has outsourced most of its electronics supply chain overseas. From the fabrication of silicon chips, to the injection molding of plastic cases, to the assembly of smartphones, it happens overseas, with several essential links going through or influenced by China. Thus weaponizing the electronics supply chain is akin to fighting a war where bullets and breeches are sourced from your enemy. Victory is not inconceivable in such a situation, but it requires planning and precision to ensure that the first territory captured in the war hosts the factories that supply your base of power.

Using the global supply chain as a weapon is like launching a missile where your enemy controls the guidance systems: you can point it in the right direction, but where it goes after launch is out of your hands. Some of the first casualties of this trade war will be the American businesses that traded with Huawei. And if China chooses to reciprocate and limit US access to its supply chain, the US could take a hard hit.

Unintended Consequences: How Weaponized Trade Could Backfire And Weaken US Tech Leadership

One of the assumed outcomes of the trade war will be a dulling of China’s technical prowess, now that its access to the best and highest performing technology has been cut off. However, unlike oil or US dollars, US dominance in technology is not inherently linked to geographic territories. Instead, the reason why the US has maintained such a dominant position for such a long time is because of a free and unfettered global market for technology.

Technology is a constant question of “make vs. buy”: do we invest to build our own CPU, or just buy one from Intel or ARM? Large customers routinely consider the option of building their own royalty-free in-house solutions. In response to such threats, US-based providers lower their prices or improve their offerings, thus swinging the position of their customers from “make” to “buy”.

Thus, large players are rarely without options when their technology suppliers fail to cooperate. Huge companies routinely groom internal projects to create credible hedge positions that reduce market prices for acquiring various technologies. It just so happens the free market has been very effective at dissuading the likes of Huawei from investing the last hundred million dollars to bring those internal projects to market: the same market forces that drove the likes of the DEC Alpha and Sun Sparc CPUs to extinction have also kept Huawei’s CPU development ambitions at bay.

The erection of trade barriers disrupts the free market. Now, US companies will no longer feel the competitive pressure of Huawei, causing domestic prices to go up while reducing the urgency to innovate. In the meantime, Huawei will have no choice but to invest that last hundred million dollars to bring a solution to market. This in no way guarantees that Huawei’s ultimate solution will be better than anything the US has to offer, but one would be unwise to immediately dismiss the possibility of an outcome where Huawei, motivated by nationalism and financially backed by the Chinese government, might make a good hard swing at the fences and hit a home run.

The interest in investing in alternative technologies goes beyond Huawei. Before the trade war, hardly anyone in the Chinese government had heard about RISC-V, an open-source alternative to Intel and ARM CPUs. Now, my sources inform me it is a hot topic. While RISC-V lags behind ARM and Intel in terms of performance and maturity, one key thing it had been lacking is a major player to invest the money and manpower it takes to close the gap. The deep irony is that the US-based startup attempting to commercialize RISC-V – SiFive – will face strong headwinds trying to tap the sudden interest of Chinese partners like Huawei directly, given the politics of the situation.

Collateral Damage: Open Source

The trade war also begs a question about the fate of open source as a whole. For example, according to the 2017 Linux Foundation report, Huawei was a Platinum sponsor of the Linux Foundation – contributing $500,000 to the organization – and they were responsible for 1.5% of the code in the Linux kernel. This is more influence than Facebook, more than Texas Instruments, more than Broadcomm.

Because the administrative action so far against Huawei relies only upon export license restrictions, the Linux Foundation has been able to find shelter under a license exemption for open source software. However, should Huawei be designated as a “foreign adversary” under EO13873, it greatly expands the scope of the ban because it prohibits transactions with entities under the direction or influence of foreign adversaries. The executive order also broadly includes any information technology including hardware and software with no exemption for open source. In fact, it explicitly states that “…openness must be balanced by the need to protect our country against critical national security threats”. While the context of “open” in this case refers to an “investment climate”, I worry the text is broad enough to easily extend its reach into open source technologies.

There’s nothing in Github (or any other source-sharing platform) that prevents your code from being accessed by a foreign adversary and incorporated into their technological base, so there is an argument that open source developers are aiding and abetting an enemy by effectively sharing technology with them. Furthermore, in addition to considering requests to merge code from a technical standpoint, one has to also consider the possibility that the requester could be subject to the influence of Huawei, in which case accepting the merge may put you at risk of stiff penalties under the IEEPA (up to $250K for accidental violations; $1M and 20 years imprisonment for willful violations).

Hopefully there are bright and creative lawyers working on defenses to the potential issues raised by EO13873.

But I will say that ideologically, a core tenant of open source is non-discriminatory empowerment. When I was introduced to open source in the 90’s, the chief “bad guy” was Microsoft – people wanted to defend against “embrace, extend, extinguish” corporate practices, and by homesteading on the technological frontier with GNU/Linux we were ensuring that our livelihoods, independence, and security would never be beholden to a hostile corporate power.

Now, the world has changed. Our open source code may end up being labeled as enabling a “foreign adversary”. I never suspected that I could end up on the “wrong side” of politics by being a staunch advocate of open source, but here I am. My open source mission is to empower people to be technologically independent; to know that technology is not magic, so that nobody will ever be a slave to technology. This is true even if that means resisting my own government. The erosion of freedom starts with restricting access to “foreign adversaries”, and ends with the government arbitrarily picking politically convenient winners and losers to participate in the open source ecosystem.

Freedom means freedom, and I will stand to defend it.

Now that the US is carpet-bombing Huawei’s supply chain, I fear there is no turning back. The language already written into EO13873 sets the stage to threaten open source as a whole by drawing geopolitical and national security borders over otherwise non-discriminatory development efforts. While I still hold hope that the trade war could de-escalate, the proliferation and stockpiling of powerful anti-trade weapons like EO13873 is worrisome. Now is the time to raise awareness of the threat this poses to the open source world, so that we can prepare and come together to protect the freedoms we cherish the most.

I hope, in all earnestness, that open source shall not be a casualty of this trade war.

Name that Ware, June 2019

The Ware for June 2019 is shown below.

I found this particular detail to be entertaining:

Some of the RAM chips have this tiny, thin PCB wrapping around three sides. That must have been a fun assembly process to develop and debug.

Thanks again to Patricio Worthalter for contributing this ware!

Winner, Name that Ware May 2019

The Ware for May 2019 is a contact image sensor from an HP multifunction print/scanner. Congrats to jackw01 for nailing this one, email me for your prize! The array of tall, thin rectangles along the bottom of the die is the actual image sensing elements, and the circuitry immediately above it is fully covered with a metal shield to prevent misdirected incident light from affecting its operation.

Below is a slightly larger contextual image of the die, which clearly shows its unusual (long and thin) aspect ratio.

In addition to about a dozen of these image sensors, the same board is paired with a couple of other wirebonded chips. One is a pretty run-of-the-mill looking driver or deserializer chip, and the other is some TI chip codenamed “Dasher” which features a mesh structure in its top metal layer. The mesh looks similar to an anti-tamper mesh, so I’m guessing there is some magic inside that HP was looking to protect from would-be cloners. I thought about making the die shots below the feature for last month’s Name That Ware, but decided it would be a bit too hard to guess, as googling for SN67013 returns quite a red herring, a speech controller chip by an unrelated company.

Name that Ware May 2019

The Ware for May 2019 is shown below.

It’s always tough to calibrate how much of a chip to show to make it identifiable, but not too easy to guess at the same time. Let’s see how this one goes…

Winner, Name that Ware April 2019

The Ware for April 2019 is a roto-molding machine. I’ve seen these machines in various sizes, but this is by far the biggest one I’ve seen to date. It’s used to mold furniture-sized objects. One thing that you don’t get from just watching an animated GIF is feeling the sheer amount of heat coming from the machine. The whole assembly is shoved into a gas furnace and heated so that the plastic is molten, and then taken out to cool over a period of 30-45 minutes, so it can get quite hot around the molds as they cool. Adi guessed this one right away, congrats and email me for your prize!

Flex PCB Fabrication

I’ve gotten a few people asking me where I get my flex PCBs fabricated, so I figured I’d make a note here. I get my flex PCBs (and actually most of my PCBs, except laser-drilled microvia) done at a medium-sized shop in China called King Credie. Previously it was a bit hard to talk about them because they only took orders via e-mail and in Chinese, but they recently opened an English-friendly online website for quotation and order placement. There’s still a few wrinkles in the website, but for a company whose specialty is decidedly not “web services” and with English as a second language, it’s usable.

Knowing your PCB vendor is advantageous for a boutique hardware system integrators like me. It’s a bit like the whole farm-to-table movement — you get better results when you know where your materials are coming from. I’ve probably been working with King Credie for almost a decade now, and I try to visit their facility and have drinks with the owner on a regular basis. I really like their CEO, he’s been a circuit board fabrication nerd since college, and he’s living his dream of building his own factory and learning all he can about interesting and boutique PCB processes.

I like to say the shop is “just the right size” for someone like me — not so big I get lost in the system, not so small that it lacks capability. Their process offering is pretty diverse for a shop their size. In addition to flex PCB, they can do multi-layer flex, rigi-flex, metal cores (for applications that require built-in heatsinking like high power LEDs), RF laminates, and laminated EMI shielding films. They can also do a variety of post-processing, such as edge plating, depth-routing, press-fit holes, screen-printed carbon and custom soldermask and silkscreen colors.

If you’re new to flexible PCBs, check out their FPC stackup page for how to set up your design tool. Flexible and rigi-flex PCBs literally open a new dimension over traditional flat PCB designs — it’s a lot of fun to design in flex!

P.S. I was not paid to write this blog. It’s just that now that King Credie has an English website, I can finally answer the question of “where do you get your PCBs fabricated” with a better answer than “there’s this factory in China … but it’s all in Chinese, so never mind”.

Name that Ware, April 2019

The ware for April 2019 is shown below:

May came up on me quick this year! Happy labour day to most of the world.