Andrew Huang's Blog, page 25

Winner, Name that Ware February 2018

The ware for February 2018 is an Ethernet card by Digitel, a Brazilian manufacturer, circa 1992. Brazil is an interesting market because protectionist trade measures made import electronics very expensive. The nominal theory, as it was explained to me, was to protect and encourage local industries, thus creating and maintaining high-paying local jobs. I had never seen a piece of electronics from Brazil, but indeed, many of the circuit board’s components bear company logos I had never seen before and a Brazilian country of origin. While at least facially it seems the trade policies created local jobs, a comparison of this card to its contemporaries outside Brasil — such as this 1992-vintage SMC “Elite 16” Ethernet card featured at vintagecomputer.net — gives a hint at how these policies might have also impeded the progress of technology.

While 0x3d named the ware almost immediately, I really appreciated the cultural insight that Paulo Peres shared about the ware. For example, the fact that the ROM labeled MAQUEST is probably “MAQuina de ESTado” (state machine) and could have been a hack at the time to use locally-produced components to substitute for imported components. Even though in a free market a ceramic EEPROM + 74-series registers would be much more expensive than a PAL, the fact that the EEPROM and registers were produced in Brazil would have made the combo cheaper than an imported PAL once the trade tariffs were factored in. So congrats Paulo, email me for your prize! Although, my understanding is the trade barriers are still in place to this day, so maybe sending you something from overseas would cost you much more duty than it’s worth if you’re located in Brazil… :-O

An Intuitive Motor: IQ Control’s Serial-to-Position Module

Back when I was a graduate student, my advisor Tom Knight bestowed upon me many excellent aphorisms. One of them was, “just wrap a computer around it!” – meaning, rather than expending effort to build more perfect systems, wrap imperfection-correcting computers around imperfect systems.

An everyday example of this is the noise-cancelling headphone. Headphones offer imperfect noise cancellation, but by “wrapping a computer around it” – adding one or more microphones and a computer in the from of a digital signal processor (DSP) – the headphones are able to measure the ambient noise and drive the headphones with the exact inverse of the noise, thus cancelling out the surrounding noise and creating a more perfect listening experience.

Although the principle has found its way rapidly into consumer goods, it’s been very slow to find its way onto the engineer’s workbench. It’s the case of the cobbler’s children having no shoes.

In particular, it’s long bothered me that motors are so dumb. Motors are typically large, heavy, costly, power-hungry, and riddled with small mechanical imperfections. In comparison, microcontrollers are tiny, cheap, power-efficient, and could run software that trims imperfections while improving efficiency to the point where the motor + microcontroller combo is a win over a dumb motor on almost every metric. So why aren’t we wrapping a computer around every motor and just calling it a day?

Then one day a startup called IQ Motion Control showed me a demo of their smart motor, the IQ Position Module, at HAX in Shenzhen. My eyes instantly lit up – these guys have done it, and done it in a tasteful manner. This is the motor I’ve been waiting years for!

Meet the IQ Position Module

Simply put, the IQ Position Module is a brushless DC motor that talks serial and “thinks” at a higher level. I don’t have to design any complicated drive circuitry or buy a proprietary controller that talks some arcane or closed standard. I just plug an FTDI cable into my laptop, hook up power, clone a small git repo and I’m good to go.

Because of the microcontroller on the inside, the IQ Position Module can emulate a range of behaviors, from a simple stepper to a range of BLDC drive standards, but the real magic happens when you tell it where you want it to go and how fast, and it figures out the best way to get there.

“But wait”, you say, “my servos and brushed DC motors can do that just fine, I just control the pulse width!” This is true for crude and slow motion control applications, but if you really want to run at high speeds – like the ones achievable by a BLDC – you have to consider things like acceleration and deceleration profiles.

The video below shows what I mean. Here is a pre-production IQ Position Module that’s being commanded to turn once in two seconds; then twice, three times, and finally ten times in two seconds. The motor can go even faster, but the figurine I attached on top isn’t balanced well enough to do that safely. Notice how the speed “ramps up” and back down again, so that the motor stops with the figurine in precisely the same position at the end of every cycle, regardless of how fast I commanded the motor to turn.

https://bunniefoo.com/bunnie/iq_turn_demo.mp4

That is magic.

And here’s a snippet of the core code used in the above demo, to give you an idea of how simple the API can be:

Just tell it where you want it, and by when — and the motor figures out an acceleration profile. Of course other parameters can be tweaked but the default behavior is reasonable enough!

A Motor That’s Also an Input Device

But wait! There’s more. Because this is a “direct drive” system, there’s no gears to shear. Anyone who has busted a geared servo motor by stalling or back-driving it knows what I mean. IQ Position Modules don’t have this problem. When you stop driving the IQ module – put it in a “coast” mode – it turns freely and without resistance.

This means the IQ motor doesn’t just “write” motion – it can “read” motion as well. Below is a video of a simple motion copy demo I cooked up in about an hour (including time spent refactoring the original API), where I implement bidirectional read/write of motion between two IQ Position Modules.

https://bunniefoo.com/bunnie/iq_motion_copy_demo_lbr.mp4

The ability to tolerate back-drive and also “go limp” is advantageous in robotics applications. Impact-oriented tasks — such as hammering a nail or kicking a ball — would rapidly degrade the teeth in a geared drive train. Furthermore, natural human motion incorporates the ability to go limp, such as the forward swing of a leg during walking. Finally, biological muscles are capable of applying a static force without changing position, such as when holding a cup on its sides without crushing it. Roboticists have developed a wide range of specialty actuators and techniques, including series elastic and variable stiffness actuators to address these scenarios. However, these mechanisms are often complicated and pricey.

The IQ Position Module’s lack of gearing means it’s back-drive tolerant, and it can apply an open-loop force without any risk of damage. This means you could, for example, use it to build a robot arm that can hammer a nail or pick up a cup. Robotic elements built using these would have far greater resilience to motion interference and impact forces than ones built using geared servos.

Having Fun with the IQ Position Module

While attending 34C3 back in December 2017, I managed to sit down for about an hour with my good friends Prof. Nadya Peek and Ole Steinhauer, and we built a 2-axis robot arm that could do kinesthetic learning through keyframing, using nothing more than two IQ Position Modules, a Dunkin’ Donuts box, a bunch of schwag stickers we stole from the FOSSASIA assembly, and the base plate of an old PS4 … because fail0verflow.

This was improvisational making at its best; we didn’t really plan the encounter so much as it emerged out of the chaos that is the Computer Congress. While Nadya was busy cutting, folding, and binding the cardboard into a 2-axis robot arm, Ole “joined” (#lötwat?!) together the power & serial connectors, and I furiously wrote the code that would do the learning and playback — while also doing my best to polish off a couple beers. Nadya methodically built one motion axis first, and we tested it; satisfied with the result, she built and stacked a second axis on top. With just a bit of tweaking and prodding we managed to pull off the demo below:

https://bunniefoo.com/bunnie/iq_2axis_robot.mp4

It’s a little janky, but given the limited materials and time frame for execution, it hints at the incredible promise that IQ Position Modules hold.

So, if you’ve ever wanted to dabble in robotics or motion control, but have been daunted by control theory and arcane driver protocols (like I’ve been), check out the IQ Position Module. They are crowdfunding now at CrowdSupply. I backed their campaign to reserve a few more Position Modules for my lab – by wrapping a smart computer around a dumb motor, they’ve created a widget that lets me go from code to physical position and back with a minimal amount of wiring and an accessible API.

Their current funding campaign heavily emphasizes the capabilities of their motor as a “better BLDC” for the lucrative drone market, and I respect their wisdom in focusing their campaign message around a single, economically significant vertical. A cardinal sin of marketing revolutionary tech is to sell it as a floor wax and a dessert topping — as painful as it may be, you have to pick just one message and push hard around it. However, I’m happy they are offering the IQ Position Module as part of the campaign, and enabling me to express my enthusiasm to the maker and robotics communities. I’ve waited too long to have a motor with this capability in my toolbox — finally, the cobbler’s children has shoes!

Name that Ware, February 2018

The Ware for February 2018 is shown below.

Ware courtesy of Hernandi Krammes!

Every board designer leaves a thumbprint on their ware — and this one is from a region I had never previously seen a ware from before. So while probably easy to guess the function, I still appreciated it for the small, unique details.

Winner, Name that Ware January 2018

The Ware for January 2018 is a front panel VFD/switch controller board for an HP Laserjet 4+. Archels nailed it — I checked u19pb1996 in Google for hits and nothing came up, but maybe I was too hasty and typo’d the number when cross-checking the image. Anyways, this post is now the top hit for that part number :) Congrats, email me for your prize.

+1 to zebonaut’s comment about the firmware code from the early 90’s never needing an update, ever…they just don’t write software like they used to anymore! It’s funny to see the panic in the eyes of a modern software developer when you tell them a subsystem has no firmware update path, ever, and their code just has to work reliably from day one. And then you tell a hardware developer the same thing and they go “yah, so?”…there’s no such thing as a downloadable hardware update, of course the product ships complete, working & tested. And not only does the hardware have to work, it carries a warranty, unlike most software…

When More is Less: China’s Perception of the iPhone X “Notch”

I recently saw a Forbes article citing rumors that the iPhone X is being cancelled this summer. Assuming the article is correct, it claims that a “lack of interest in China” is the main reason for the relatively early cancellation of production. They’re hoping that 6.1″ and 6.5″ versions of their phone with a less pronounced Face ID notch would excite Chinese customers.

The notion of a “less pronounced” Face ID notch is what got me — Apple embracing the notch as iconic, and worth carrying forward to future models, rather than simply making the top bezel a bit larger and eliminating the notch altogether. Historically, Apple has taken a “less is more” strategy, meticulously eliminating even the tiniest design facets: replacing radii with splines, polishing off injection mold parting lines, even eliminating the headphone jack. Putting a notch on the iPhone feels a bit like watching a woman painstakingly apply face whitening cream day after day to remove tiny blemishes, and then don a red clown nose.

Like the red clown nose, the problem with pushing the notch is that anyone can put one on, should they decide it’s a feature they want to copy. Xobs recently showed me an app on his Xiaomi Mix 2 that does exactly that. Below is what his Xiaomi Mix 2 looks like normally.

It’s got a screen that goes right up to the top bezel, without a notch.

Interestingly, there’s an app you can run called “X out of 10” that simply draws in the notch (including subtle details like a simulated camera lens). Here’s the app in the off state:

And now in the on state:

Once activated and given permission to draw over other apps, Xiaomi Mix 2 users can don the red clown nose and experience the full glory of the iconic Apple notch all the time:

This glass-half-empty situation is a parable for design leadership versus market perception: if a market previously lacked a smartphone with a minimal top bezel, the notch is perceived as “How innovative! I’ve got extra pixels to the left and right of my earpiece/camera assembly!”. But once a market has seen a smartphone with minimal top bezel, the notch turns into “Hey where did my pixels go? What’s this notch doing here?”. It’s a case where the additional design feature is seen as a loss of function, not a gain.

Thus it will be interesting to see if Apple’s bet to introduce a phone with a larger screen that can compete head to head in China against the likes of the Xiaomi Mix 2’s 6″ screen will pay out, especially if Apple retains the notch.

Of course, as the design space for phones becomes more and more crowded, Apple’s room to maneuver becomes increasingly limited. The minimalist design space is winner-takes-all: the first company to elegantly remove a design facet wins the minimalism race, and now that Xiaomi has planted a flag in the bezel-less top space, it may be that Apple has no option but to sport the top-notch, or run the risk of being seen as copying a Chinese company’s design language.

ICE40 for Novena

For any and all Novena users, a quick note: Philipp Gühring is organizing a production run of ICE40 FPGA add-in cards for people who want a 100% open software stack for making FPGA bitfiles. Register your interest in the production run by visiting his website!

Spectre/Meltdown Pits Transparency Against Liability: Which is More Important to You?

There is a lot of righteous anger directed toward Intel over CPU bugs that were revealed by Spectre/Meltdown. I agree that things could have been handled better, particularly with regards to transparency and the sharing of information among the relevant user communities that could have worked together to deploy effective patches in a timely fashion. People also aren’t wrong that consumer protection laws obligate manufacturers to honor warranties, particularly when a product is not fit for use as represented, if it contains defective material or workmanship, or fails to meet regulatory compliance.

However, as an open source hardware optimist, and someone who someday aspires to see more open source silicon on the market, I want to highlight that demanding Intel return, exchange, or offer rebates on CPUs purchased within a reasonable warranty period is entirely at odds with demands that Intel act with greater transparency in sharing bugs and source code.

Transparency is Easy When There’s No Penalty for Bugs

It’s taken as motherhood and apple pie in the open source software community that transparency leads to better products. The more eyes staring at a code base, the more bugs that can be found and patched. However, a crucial difference between open source software and hardware is that open source software carries absolutely no warranty. Even the most minimal, stripped down OSS licenses stipulate that contributors carry no liability. For example, the BSD 2-clause license has 189 words, of which 116 (60%) are dedicated to a “no warranty” clause – and all in caps, in case you weren’t paying attention. The no-warranty clause is so core to any open source license it doesn’t even count as a clause in the 2-clause license.

Of course contributors have no liability: this lack of liability is fundamental to open source. If people could sue you for some crappy code you you pushed to github years ago, why would you share anything? Github would be a ticking time bomb of financial ruin for every developer.

It’s also not about code being easier to patch than hardware. The point is that you don’t have to patch your code, even if you could. Someone can file a bug against you, and you have the legal right to ignore it. And if your code library happens to contain an overflow bug that results in a house catching fire, you walk away scot-free because your code came with no warranty of fitness for any purpose whatsoever.

Oohh, Shiny and New!

Presented a bin of apples, most will pick a blemish-free fruit from the bushel before heading to the check-out counter. Despite the knowing the reality of nature – that every fruit must grow from a blossom under varying conditions and hardships – we believe our hard-earned money should only go toward the most perfect of the lot. This feeling is so common sense that it’s codified in the form of consumer protection laws and compulsory warranties.

This psychology extends beyond obvious blemishes, to defects that have no impact on function. Suppose you’re on the market to buy a one-slot toaster. You’re offered two options: a one-slot toaster, and a two-slot toaster but with the left slot permanently and safely disabled. Both are exactly the same price. Which one do you buy?

Most people would buy the toaster with one slot, even though the net function might be identical to the two-slot version where one slot is disabled. In fact, you’d probably be infuriated and demand your money back if you bought the one-slot toaster, but opened the box to find a two-slot toaster with one slot disabled. We don’t like the idea of being sold goods that have anything wrong with them, even if the broken piece is irrelevant to performance of the device. It’s perceived as evidence of shoddy workmanship and quality control issues.

News Flash: Complex Systems are Buggy!

Hold your breath – I’d wager that every computer you’ve bought in the past decade has broken parts inside of them, almost exactly like the two-slot toaster with one slot permanently disabled. There’s the set of features that were intended to be in your chips – and there’s the subset of series of features that finally shipped. What happened to the features that weren’t shipped? Surely, they did a final pass on the chip to remove all that “dead silicon”.

Nope – most of the time those partially or non-functional units are simply disabled. This ranges from blocks of cache RAM, to whole CPU cores, to various hardware peripherals. Patching a complex chip design can cost millions of dollars and takes weeks or even months, so no company can afford to do a final “clean-up” pass to create a “perfect” design. To wit, manufacturers never misrepresent the product to consumers – if half the cache was available, the spec sheet would simply report the cache size as 128kB instead of 256kB. But surely some customers would have complained bitterly if they knew of the defect sold to them.

Despite being chock full of bugs, vendors of desktop CPUs or mobile phone System on Chips (SoCs) rarely disclose these bugs to users – and those that do disclose almost always disclose a limited list of public bugs, backed by an NDA-only list of all the bugs. The top two reasons cited for keeping chip specs secret are competitive advantage and liability, and I suspect in reality, it’s the latter that drives the secrecy, because the crappier the chipset, the more likely the specs are under NDA. Chip vendors are deathly afraid users will find inconsistencies between the chip’s actual performance and the published specs, thus triggering a recall event. This fear may seem more rational if you consider the magnitude of Intel’s FDIV bug recall ($475 million in 1994).

This is a pretty typical list of SoC bugs, known as “errata”. If your SoC’s errata is much shorter than this, it’s more likely due to bugs not being disclosed than there actually being less bugs.

If you Want Messages, Stop Shooting the Messengers

Highly esteemed and enlightened colleagues of mine are strongly of the opinion that Intel should reimburse end users for bugs found in their silicon; yet in the same breath, they complain that Intel has not been transparent enough. The point that has become clear to me is that consumers, even open-source activists, are very sensitive to imperfections, however minor. They demand a “perfect” machine; if they spend $500 on a computer, every part inside better damn well be perfect. And so starts the vicious cycle of hardware manufacturers hiding all sorts of blemishes and shortcomings behind various NDAs, enabling them to bill their goods as perfect for use.

You can’t have it both ways: the whole point of transparency is to enable peer review, so you can find and fix bugs more quickly. But if every time a bug is found, a manufacturer had to hand $50 to every user of their product as a concession for the bug, they would quickly go out of business. This partially answers the question why we don’t see open hardware much beyond simple breakout boards and embedded controllers: it’s far too risky from a liability standpoint to openly share the documentation for complex systems under these circumstances.

To simply say, “but hardware manufacturers should ship perfect products because they are taking my money, and my code can be buggy because it’s free of charge” – is naïve. A modern OS has tens of millions of lines of code, yet it benefits from the fact that every line of code can be replicated perfectly. Contrast to a modern CPU with billions of transistors, each with slightly different electrical characteristics. We should all be more surprised that it took so long for a major hardware bug to be found, than the fact that one was ever found.

Complex systems have bugs. Any system with primitives measured in the millions or billions – be it lines of code, rivets, or transistors – is going to have subtle, if not blatant, flaws. Systems simple enough to formally verify are typically too simple to handle real-world tasks, so engineers must rely on heuristics like design rules and lots and lots of hand-written tests.

There will be bugs.

Realities of the Open Hardware Business

About a year ago, I had a heated debate with a SiFive founder about how open they can get about their documentation. SiFive markets the RISC-V CPU, billed as an “open source CPU”, and many open source enthusiasts got excited about the prospect of a fully-open SoC that could finally eliminate proprietary blobs from the boot chain and ultimately through the same process of peer review found in the open source software world, yield a more secure, trustable hardware environment.

However, even one of their most ardent open-source advocates pushed back quite hard when I suggested they should share their pre-boot code. By pre-boot code, I’m not talking about the little ROM blob that gets run after reset to set up your peripherals so you can pull your bootloader from SD card or SSD. That part was a no-brainer to share. I’m talking about the code that gets run before the architecturally guaranteed “reset vector”. A number of software developers (and alarmingly, some security experts) believe that the life of a CPU begins at the reset vector. In fact, there’s often a significant body of code that gets executed on a CPU to set things up to meet the architectural guarantees of a hard reset – bringing all the registers to their reset state, tuning clock generators, gating peripherals, and so forth. Critically, chip makers heavily rely upon this pre-boot code to also patch all kinds of embarrassing silicon bugs, and to enforce binning rules.

The gentleman with whom I was debating the disclosure of pre-boot code adamantly held that it was not commercially viable to share the pre-boot code. I didn’t understand his point until I witnessed open-source activists en masse demanding their pound of flesh for Intel’s mistakes.

As engineers, we should know better: no complex system is perfect. We’ve all shipped bugs, yet when it comes to buying our own hardware, we individually convince ourselves that perfection is a reasonable standard.

The Choice: Truthful Mistakes or Fake Perfection?

The open source community could use the Spectre/Meltdown crisis as an opportunity to reform the status quo. Instead of suing Intel for money, what if we sue Intel for documentation? If documentation and transparency have real value, then this is a chance to finally put that value in economic terms that Intel shareholders can understand. I propose a bargain somewhere along these lines: if Intel releases comprehensive microarchitectural hardware design specifications, microcode, firmware, and all software source code (e.g. for AMT/ME) so that the community can band together to hammer out any other security bugs hiding in their hardware, then Intel is absolved of any payouts related to the Spectre/Meltdown exploits.

This also sets a healthy precedent for open hardware. In broader terms, my proposed open hardware bargain is thus: Here’s the design source for my hardware product. By purchasing my product, you’ve warranted that you’ve reviewed the available design source and decided the open source elements, as-is, are fit for your application. So long as I deliver a product consistent with the design source, I’ve met my hardware warranty obligation on the open source elements.

In other words, the open-source bargain for hardware needs to be a two-way street. The bargain I set forth above:

Rewards transparency with indemnity against yet-to-be-discovered bugs in the design source
Burdens any residual proprietary elements with the full liability of fitness for purpose
Simultaneously conserves a guarantee that a product is free from defects in materials and workmanship in either case

The beauty of this bargain is it gives a real economic benefit to transparency, which is exactly the kind of wedge needed to drive closed-source silicon vendors to finally share their full design documentation, with little reduction of consumer protection.

So, if we really desire a more transparent, open world in hardware: give hardware makers big and small the option to settle warranty disputes for documentation instead of cash.

Author’s Addendum (added Feb 2 14:47 SGT)

This post has 2 aspects to it:

The first is whether hardware makers will accept the offer to provide documentation in lieu of liability.

The second, and perhaps more significant, is whether you would make the offer for design documentation in lieu of design liability in the first place. It’s important that companies who choose transparency be given a measurable economic advantage over those who choose obscurity. In order for the vicious cycle of proprietary hardware to be broken, both consumer and producer have to express a willingness to value openness.

Name that Ware, January 2018

The Ware for January 2018 is shown below.

This side of the board might be a little too non-descript to make a solid guess, so if nobody gets it within a couple of weeks, I’ll push a picture of the other side.

Thanks to spida for handing me pictures of this well-photographed ware at 34C3!

Winner, Name that Ware December 2017

The winner of Name that Ware Dec 2017 is Piotr! Congrats, email me for your prize. He nailed it very quickly as a 2000-series multimeter by Keithley. I figured since this is a classic meter found on many engineer’s benches, it would be named fairly quickly, even with a tightly cropped shot of the main board.

My decades-old Keithley 2000 finally gave up the ghost; something seems wrong in the input stage that’s preventing it from both doing autoranging and measuring negative voltages. A quick thermal scan showed a few transistors getting hot that shouldn’t be, and the self-test codes indicate that something may have gone very wrong with the core ADC pathway. I’ve set it aside to look at later and ended up picking up a Keithley 2110 as a stand-in replacement until I can afford a DMM7510. There’s some pretty nifty debugging tricks you can do with a 7.5-digit multimeter (it’s sensitive enough to differentiate between code execution paths on an MCU, or observe a battery’s self-discharge rate in real time), but at the moment I simply can’t afford one. Ah well — it’s always good to have goals!

Name that Ware December 2017

The Ware for December 2017 is shown below.

I’ve partially cropped the photo to make it a bit more challenging, but I have a feeling this will be guessed … rather quickly … despite the impediment.

It’s a real beauty on the inside, isn’t it?