Toby Segaran's Blog, page 3

Tuxedo Jacket

I decided I wanted to make something a little more elegant for Halloween this year (and for future parties / Burning Man). It’s a little hard to capture on video, but it’s really quite mesmerizing:

This is a tuxedo jacket that I bought at a thrift shop on Valencia St. ($17 for a Dior jacket that fit perfectly!) I ran WS2811 strips along the seams so as to take advantage of the natural lines of the jacket. They are wired together on the inside. The first strip terminates right at the bottom of the jacket and has three wires running up the inside to the front pocket.

In the pocket is the battery pack (4 Lithium AAs) and an Arduino Pro Micro. The Arduino controls the lights with the Adafruit Neopixel library. I put a toggle switch between the batteries and the Arduino so I could plug it into a USB port for programming while still running the light strips off the batteries.

This is the first program I wrote for it, and I’m pretty happy. I’m really excited that I can easily program dozens of patterns and also add sensors to it (motion, sound, light/color, etc.) Expect to see more videos of the jacket in the future.

Twitter lights and memory limits with Arduino Yun

When the Arduino YÚN was announced, I was definitely curious. Many people said it was overpriced because a Raspberry Pi with all the necessary add-ons was slightly cheaper and more powerful.  Although I’ve managed to do some interesting things with my Pi (more on my home automation system to come), I’ve definitely found that it’s always been a little faster to get an Arduino interfacing with external components and they’re much harder to break. Rather than explain what it’s all about (which you can find elsewhere),

I got my Yun a few days ago, delivered at work. When I got home I figured I had about an hour to play with it before I had to go meet some people for drinks. I already had soldered this colorful shield and was familiar with the Neopixel libraries so I had an option for display. I was most interested in setting it up to read some information from the web via wifi and having it change the colors on the shield to reflect some kind of updates.

The Yun comes preloaded with libraries for Temboo, which saves a lot of coding on the Linux side, and just lets you write regular Arduino sketches. They had an example for Twitter, so I decided to go with that and see if I could show my Twitter feed as a sequence of colors on the Neopixels.

If you read the example, you’ll notice that (a) it uses “string” which I hadn’t really seen in Arduino code before (and it’s not the usual C string I’m used to) and (b) it only retrieves one tweet. It took some effort to figure out how to make Temboo retrieve more than one tweet. You’d think you could just update this line:

HomeTimelineChoreo.addInput("Count", "1");

Unfortunately, changing this to more doesn’t work because these lines:

HomeTimelineChoreo.addOutputFilter("tweet", "/[1]/text", "Response");
HomeTimelineChoreo.addOutputFilter("author", "/[1]/user/screen_name", "Response");

Only fill one of the variables no matter how many you retrieve. I created a loop which fills in the filters the way Temboo expects them though.

The next problem I had was although I was trying to retrieve 40 tweets (the number of lights on the shield), I was only getting 7 back. It took me a while to understand why, but then it occurred to me — the ATMega was running out of RAM. Since I’d never actually tried text processing on a microprocessor this small before, I hadn’t expected this.

So I removed the line that retrieved the tweets and just retrieved the author names. This allowed me to get about 15-20 results, but still not enough to fill all my pixels. As a guess, I assumed that Temboo was able to get all the results (the Linux side having a lot more RAM), but they got packaged and sent to the Arduino which didn’t have enough RAM to handle it. To work around this, I kept the count at 40 but ran the request 4 times and had a different set of output filters each time (the first 10, the second 10, etc.). And it worked!

(if you’ve tried to photograph fully lit LEDs before you’ll understand that this photo was actually taken in daylight and the shield is difficult to look directly at)

Each light represents a tweet, and the color is generated from the first letter of the authors username. Once I realized the pitfalls, it was pretty easy to get something cool built in a little over an hour (although when debugging a twitter app, hitting rate limits is a problem)

Here’s the code if you’re curious. Notice how the same thing is retrieved 4 times but with different output filters:

#include
#include
#include
#include "TembooAccount.h" // contains Temboo account information
                           // as described in the footer comment below

// Note that for additional security and reusability, you could
// use #define statements to specify these values in a .h file.
const String TWITTER_ACCESS_TOKEN = "";
const String TWITTER_ACCESS_TOKEN_SECRET = "";
const String TWITTER_CONSUMER_KEY = "";
const String TWITTER_CONSUMER_SECRET = "";

int numRuns = 1;   // execution count, so this doesn't run forever
int maxRuns = 10;   // the max number of times the Twitter HomeTimeline Choreo should run

Adafruit_NeoPixel strip = Adafruit_NeoPixel(40, 6, NEO_GRB NEO_KHZ800);

void setup() {
  strip.begin();
  strip.show(); 
  delay(4000);
  Bridge.begin();
}
void loop()
{
  for (int outer=0;outer< 85) {
   return strip.Color(WheelPos * 3, 255 - WheelPos * 3, 0);
  } else if(WheelPos < 170) {
   WheelPos -= 85;
   return strip.Color(255 - WheelPos * 3, 0, WheelPos * 3);
  } else {
   WheelPos -= 170;
   return strip.Color(0, WheelPos * 3, 255 - WheelPos * 3);
  }
}

If you have questions about the code, put them in the comments so I can answer them for everyone!

Follow @kiwitobes

Disco Stick

For New Years 2013, my friend Steve Davis organized a big party in Geyserville at a very strange venue with lots of Ocelots. I had an old HL1606 light strip that I’d bought from Adafruit so I thought it might be interesting to make something for the party.

Unlike newer light strips (the new WS2812 integrated strips are awesome), the HL1606 doesn’t support PWM, so there are a limited range of colors and they all run at full brightness. So while normally I might change the intensity of the lights with the power in various frequency bands, in this case I decided to go with more of graphic equalizer thing:

I particularly like how easy it is to see the different frequencies hit. Also, it’s really bright. As always, it’s hard to film LEDs at full brightness, but this definitely lights up the room.

Follow @kiwitobes

Sound reactive GE Christmas Lights

A couple of years ago, I found some posts online about hacking the protocol that controlled GE Christmas lights. Combined with an Arduino and an MSGEQ7 (along with some posterboard and a bladed tool that cuts circular holes) I made this:

That was fun and people thought it was neat, but there are always more interesting things to do than just a grid. I experimented with different structures, eventually building a fairly large acrylic tetrahedron:

I brought this to a dance party, which people really enjoyed, and where I was referred to as an “artist” for the first time. I experimented with other shapes, which I don’t have videos of. The Christmas lights and the MSGEQ7 ended up being pretty cool tools that I was able to use in a lot of later projects.

Follow @kiwitobes

A hack journal

I have a lot of hobbies and enjoy experimenting with electronics, food, cocktails, party concepts, chemistry and physical feats. Sometimes people tell me I should write this stuff down. I was never very good at blogging, but I’m trying again.

My self-hosted WordPress installation fell into disrepair and I didn’t have the patience to fix it. I tried using Google+ but it’s not really expressive enough for long posts.

So now I’m trying this…

Follow @kiwitobes

Posting on Google+

So, it’s been over a year since I last posted here. For some reason the platform never quite clicked for me. I’ve been enjoying using Google+ (disclaimer, yes I work for Google) a lot more, sharing photos, links and thoughts there.

I’m not sure I’ll post here any more, but I’ll leave it up cause some of the entries are still pretty popular.

Anyway, my Google+ page is here.

Visualizing UBS Analysts

Last November, my friend Jesper and I presented at Web 2.0 New York, a talk called Freeing and Visualizing Financial Data. I’ve been meaning to put up some of the visuals we made while looking through what was available.

Banks are required to publish a report each quarter called an “Analyst Transparency Report”. The report lists all the stocks that the bank’s analysts wrote reports about, along with who wrote the report and what their recommendation was. Because the report is just a several-hundred page table, it’s very difficult to get a view of the big picture. It’s even difficult to figure out basic stuff like all the stocks covered by a particular analyst.

To get an idea of what was going on, I took the report and turned it into a large network diagram (longtime readers will start to think that this is my answer to everything )

[image error]

(click on the graph to see a high-res version)

The rectangular nodes are analysts and the oval nodes are stocks. The connections between them indicate that the recommendation was buy (green), neutral (black) or sell (red).

You can spend a long time looking at this and noticing patterns that emerge. One of the cool things is that it’s easy to quickly spot companies that appeared together in a report so you can tell that they potentially have related fortunes. Here’s a great example:

[image error]

In this image, you can see that this analyst, Kevin Crissey, seems to specialize in the travel industry. He covers, for example both JetBlue (JBLU) and Expedia (EXPE). What’s fantastic about this is that the industry classification for these two companies is completely different. JBLU is classified as a Regional Airline, which EXPE is classified as General Entertainment. Because the clusters in these graphs show relationships between companies that don’t match their hierarchical classifications, they are a great source of information about potential correlations.

Here’s another section of the graph:

[image error]

What popped out to me here is that this group of three retail analysts all have recommendations on Zumiez (ZUMZ). Not only that, but two of them have written two reports and change from a neutral to sell rating. This is curious because ZUMZ is only a $600million company, which has three analysts who wrote five reports on it. There are much larger companies which have no coverage or only one analyst. Further investigation might reveal that UBS has a particular reason for all this coverage or that ZUMZ did something to attract a lot of attention.

There are many other interesting groupings and links buried in the graph. I think viewing extremely long tables as a graph like this could help us spot relationships we might not have otherwise seen.

My latest two books now available!

[image error]

The first of these is Programming the Semantic Web. I wrote this with two of my coworkers, Jamie Taylor and Colin Evans. We were attempting to make the first ever practical guide to why regular programmers should pay attention to semantic technologies. After writing this book, I was so convinced myself that I’ve moved all my projects away from traditional relational databases to graph databases.

That animal on the cover is a Red Panda, also known as a Firefox. Many thanks to our editor Mary Treseler for being so awesome through the process of writing this.

The second is Beautiful Data, which is an essay collection that I co-edited with Jeff Hammerbacher and to which I also contributed. We found a group of people who we thought were doing awesome stuff with data and convinced them to write essays.

We have an awesome list of contributors: Peter Norvig, Nathan Yau, Jonathan Follett, Matt Holm, J.M. Hughes, Raghu Ramakrishnan, Brian Cooper, Utkarsh Srivastava, Jason Dykes, Jo Wood Jeff Jonas, Lisa Sokol, Jud Valeski, Alon Halevy, Jayant Madhavan, Aaron Koblin, Valdean Klump, Michal Migurski Jeff Heer, Coco Krumme, Matt Wood, Ben Blackburne, Jean-Claude Bradley, Rajarshi Guha, Andrew Lang, Pierre Lindenbaum, Cameron Neylon, Antony Williams, Egon Willighagen, Lukas Biewald, Brendan O’Connor, Hadley Wickham, Deborah Swayne, David Poole Andrew Gelman, Jonathan P. Kastellec, Yair Ghitza and Jeff and myself.

All royalties for Beautiful Data are split between the Sunlight Foundation and Creative Commons.

Julie Steele, the O’Reilly editor on this book, was so awesome at making sure everyone got their essays in and they were reviewed properly.

They’re both great books, I’m really proud of how they turned out. I’m not sure I’ll be writing again for a while though!

Quick updates: Wedding, Hack Day, Books

I’m about to head off to my wedding in Mexico! I’m afraid while I’m gone I won’t be checking email very much, so please don’t be sad if I don’t write back to you immediately, I’ll try to get through everything when I get back!
If you like my posts on semantics and free data, and you live in San Francisco, you should check out Freebase Hack Day to meet a bunch of like-minded people.
Finally, I believe both Programming the Semantic Web and Beautiful Data will be out in July. I’m not sure of the exact date, but we’re hoping that everything is out in time for OSCON.

Why Semantics?

In February I gave a tutorial and a talk at the most awesome conference ever (go Tash!) called Webstock, in Wellington, New Zealand. The talk was called Why Semantics, and was essentially about the ideas behind the semantic web and why they’re interesting to normal working developers. After I gave the talk, I had several famous (at least to me) developers tell me that they finally got it, had made many of the data-modeling mistakes that I outlined, and no longer thought the Semantic Web was all hype.

The video was just uploaded to Vimeo by the Webstock team:

(if the embed isn’t showing, you can find the video here)

And here’s the abstract:

Ever since there was a web, people have been talking about the “semantic web”, which is always just around the corner. Even though this hasn’t exactly gone to plan, people working on the ideas behind semantic data modeling have actually come up with a lot of cool stuff.


Modern web development is very concerned with rapid iteration, which has led to the increasing popularity of lightweight frameworks built on dynamic languages such as Rails, Pylons and Django. However, most of us are still stuck using traditional data-modeling methods like relational databases which aren’t designed for constant schema changes. Further, because people don’t think about “standard” ways to share data, there are thousands of different web APIs, all of which have to be dealt with separately.


In this talk Toby will explain what “semantic data” is, how entities and data can be modeled using graphs, and show examples of modeling, integrating, and extending data models for large datasets. You’ll lean how the semantic models support rapid and iterative application development, and easy integration of existing databases. Toby will introduce fast scalable back-ends for storing and querying semantic data and show examples of semantic data already available on the web.


He’ll also briefly discuss how these approaches lead into the standards-based Semantic Web, and how attendees can find short-term value in adopting some of the Semantic Web standards and platforms.

Enjoy! Let me know what you think.

Update: You can find a PDF of the slides here.