Phil Simon's Blog, page 96

Who Needs a Big Data Thought Leader?

Five years ago, no one would have ever called me a thought leader. Sure, I knew a thing or 20 about technology, but that term just wasn’t apropos. After all, I didn’t keynote conferences, write books, blog for high-profile websites, or maintain an active social media presence. People didn’t seek out my expertise, at least beyond people on my consulting gigs.

My, how times have changed.

Fast forward to 2013, and I often get introduced to people as a thought leader. People want to me to contribute to their companies’ sites, speak at their conferences, and pick my brain (far too often without paying, but that’s a separate discussion.)

As I think about the buzz building over Big Data, it’s evident to me that many organizations and professionals are looking for guidance, advice on how to navigate an increasingly noisy and crowded terrain. In fact, I was talking about this with a friend of mine recently. The question came up, Would you work for a company on a full-time basis as a Big Data thought leader or evangelist?

It was an interesting chat and, after some back and forth, I decided that I’d be open to the right opportunity at the right company. In a way, providing thought leadership around Big Data for a specific company wouldn’t be entirely different than what I’m doing now: learning, speaking, and writing about Big Data. The only differences: I’d receive a biweekly paycheck and I couldn’t call myself independent or vendor-neutral anymore. Everything is a mixture of pros and cons, right?

There’d be conditions on both ends, to be sure. For my part, here goes:

I’d prefer to work for an agile company. I just couldn’t see being one of 30 thought leaders on Big Data in a stodgy, bureaucratic organization.
The company would have to let me honor my current writing commitments, some of which may be with potential competition. (While I don’t write “Buy Product X because it rocks” posts, I’d be naive to ignore the possibility of conflicts. I’m not about to break my existing contracts.)
Next, I would think about the job in terms of months, not years. Big Data isn’t going anywhere, but I’m not finished writing books on different topics. I’m not looking to pigeonhole myself for the next five years.

I’m really just writing this post to see what’s out there. I’m very content working for myself. My boss is pretty cool. Still, sometimes I wonder about what else is out there.

So, who’s looking for a Big Data thought leader or evangelist?

 

Acceptable Downtime: A Relic of a Bygone Era?

Last Christmas Eve, many Netflix users planned to bundle up with their loved ones and watch a bunch of streaming movies. Unfortunately, plans went awry. Netflix runs its streaming service on Amazon AWS. Because of an AWS service outage, presumably millions of Netflix customers couldn’t dial up their favorite movies and TV shows. (Some people actually had to talk to each other during the holidays. Perish the thought!) Kidding aside, the incident demonstrated a number of things and this post looks at them.

Lessons from the AWS Outage

Now, the notion of acceptable downtime has been with us for a long time. In my consulting career, I’ve seen different people and organizations cling to wildly different definitions of the term. While opinions vary, everyone agrees that downtime needs to be minimized for many reasons, not the least of which is security. From a 2009 Microsoft article:

Excessive downtime can result in increased exposure to malware, which can lead to many business losses, including the loss of sales, loss of customer goodwill, loss of productivity, loss of competitiveness, missed contractual obligations, and increased costs resulting from the need to make up these losses.

Today, there is no longer such a thing as acceptable downtime. What used to be considered acceptable is now excessive. Fifteen years ago, consumers, employees, partners, and suppliers were not constantly connected. Remember the quaint old days in which we used to get work done at, you know, work? I do. Marissa Mayer’s recent decision on banning remote work wouldn’t have caused such a kerfuffle in 1998. Relatively few of us worked remotely, especially in comparison to today. For years now, entire companies like WordPress.com and a few mentioned in The New Small have employed entirely distributed workforces. That is, employees don’t come to work. Ever.

Today, there is no such a thing as acceptable downtime.

Second, the AWS outage and subsequent Netflix backlash (yes, the topic was trending on Twitter) illustrates how reliant we are upon cloud services–whether we know it or now. Joe Consumer didn’t blame AWS or Jeff Bezos. To him, Netflix was down. End of story.

Finally, the AWS outage was the exception that proved the rule. Put differently, AWS sports up-time well north of 99 percent. When related services like Netflix stop working, it only underscores the fact that they are working the vast majority of the time.

Simon Says

Are there legitimate concerns with the public cloud? You betcha. By the same token, though, the importance of business continuity is impossible to overstate. Ours is a truly global economy now. Downloads to apps, Likes, hits to websites, and product orders take place 24/7. Against, this backdrop, organizations need to constantly minimize their downtime.

Feedback

What is your company doing to maximize up-time?

This post is sponsored by the Online VMware Forum 2013. To learn more, register for a VMWare webinar on this subject: Unleash the Power of Virtualization to Simplify IT.

The Myth of the Data Scientist

It’s hard to find a sexier job these days than data scientist. Demand far exceeds supply and has for some time now. But what if you’re one of the organizations lucky enough to land one of these coveted individuals? Are you home free? Is it only a matter of time before you start to reap massive rewards from Big Data?

In a word, no. For several reasons, even vaunted data scientists can only do so much. In this post, I’ll look at several factors that constrain many organizations that employ proper data scientists.

Data

Data scientists don’t bring with them magical tools to purify enterprise data. They don’t magically look at a datasets and de-dupe records, provide key metadata, and purge bad data with a wave of their wands. Lamentably, many organizations still don’t know how many customers, products, and employees they have.

Now, don’t get me wrong. This doesn’t mean that data scientists are impotent here. Rather, all else being equal, the better an organization’s internal data, data management practices, and governance, the more it will get out of its data scientists.

Better data equals better data scientists.

Tools

Data scientists know how to use new and sophisticated tools like R. But they don’t link themselves to data warehouses and start spitting out clusters and nodes. They don’t plug in zip drives that deploy Hadoop or columnar databases throughout the entire organization. To be successful, data scientists need the ability to access and analyze vast troves of information. If you think that your legacy data warehouse or relational database is good enough, you’re probably wrong.

Culture

This is perhaps the key ingredient in maximizing the likelihood of data scientists’ success. As my friend Melinda Thielbar recently wrote, “Great customers may not know what they need, but they’re willing to find out.” She continues:

A great customer understands that they’ve employed us because they have a problem that needs solving. They work with us to define the project scope early, and they are open to conversation during the project. A data science project always involves a bit of the unknown, and while clear expectations at the beginning are important, re-checking assumptions and explanations is vital to the process.

Spot on. Organizations afraid of the unknown are unlikely to benefit from the knowledge, skills, and perspectives that data scientists bring to the table. Historically, science has progressed steadily, but not in a linear fashion. Employees–especially senior leader–who insist upon complete certainty prior to proceeding will more often than not stifle creativity and innovation.

Simon Says

Yes, data scientists matter. Before you make the considerable investment of hiring one, however, heed the advice in this post.

Feedback

What say you?

This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don’t necessarily represent IBM’s positions,

Interview with TechCocktail on Big Data

In 2011, I did an interview with the folks from TechCocktail on The Age of the Platform. A few months ago, I had a chance to meet the folks from the company here in Las Vegas. Below you can read my second TC interview on the new book:

Tech Cocktail: What does “Big Data” mean?

Phil Simon: You’d think that such a simple term would have a commensurately simple definition, right? It doesn’t. I’ll try to simplify as much as I can.

Most people think of data as rows and columns in an Excel spreadsheet or in an Oracle database. To be sure, that structured (and often transactional) data still matters. Think lists of customers, employees, sales, etc.

Big Data represents mostly unstructured data information found in blog posts, customer reviews, call detail records, tweets, YouTube videos, podcasts, emails, and a host of other sources. Increasingly, more data is generated by machines via sensors. Think the Nest Thermostat, smart grids, and the Internet of Things. There’s no shortage of sources of data. If you aggregate them all, you wind up with Big Data.

To read the whole thing, click here.

Everything is Obvious by Duncan Watts

Duncan Watts has an interesting take on bestsellers, successful companies, and iconic pieces of art. With the benefit of hindsight, we can explain why rise above the din of other books, companies, and statues. But those explanations are much are less useful than they seem.

Everything is Obvious*: How Common Sense Fails Us (affiliate link) is an amazing book. (The asterisk signifies “Once You Know the Answer.”)

It is exactly the kind of book that makes people uncomfortable. And that’s exactly why everyone should read it. Watts shows that we don’t know nearly as much as we think we do. Written in a much more accessible style than The Black Swan: The Impact of the Highly Improbable (affiliate link). Everything is Obvious should be required reading for leaders of industry and government. In a nutshell, it shows that we don’t know nearly as much as we think we do–and why.

Yes, Big Data means that we’ll be able to increase our ability to predict and explain things. Yes,, analytics will improve, but don’t for a minute think that we’ll be able to know everything. We’ll continue to miss major trends, bubbles, revolutions, and scores of other critical events.

This is a book that I routinely recommend to my friends.

The Zen of Big Data

[image error]

I have very little doubt that many CXOs will dismiss Big Data because they realize that they can’t get all of the data. They are ignoring Big Data for many reasons. Perhaps most glaring: because they know that they are probably missing something, perhaps something big. They can’t be absolutely sure that they are analyzing and interpreting all of data–or the right data. So they demur.

Historically, many organizations have been able to capture every (internal) transaction, every check, and every journal entry. Small Data is like that. Remember, Small Data is largely internal to the enterprise. There’s a great deal of control and data governance involved–a great deal of proper data management.

Big Data is different. It’s mostly external to the enterprise. It is fundamentally unmanageable, at least in the traditional sense.

There’s no recipe for harnessing the power of Big Data, but the leaders of the companies that I researched in writing Too Big to Ignore understood the limits of their own internal controls. Yet they embraced Big Data anyway. They know that you can’t stop someone from writing a review, tweeting, uploading a video, slamming you on Facebook, etc. New data sources like Pinterest spring up seemingly overnight.

Embrace Big Data even if you can’t get all of the data.

Another Sample from Too Big to Ignore

My publisher (John Wiley & Sons) previously allowed me to make the 28-page Introduction of Too Big to Ignore available for free download. (See sidebar of this post.) In it, I discuss potholes, car insurance, and other things affected by technology and Big Data.

The book is part of the SAS Business Series. SAS has made the book’s first chapter available for download as well. Click here to read it.

Proof that Big Data Has Gone Mainstream

When Charlie Rose mutters the words “Big Data”, you know that it has arrived.

On Friday, Jeffrey Hammerbacher, Chief Scientist at Cloudera on Big Data, appeared at the revered oak table. For the research on Too Big to Ignore, I researched Hammerbacher and his company. He’s a very smart cat.

The interview is more about science than the drivers behind Big Data, but I love the focus on the practical applications of all this information. Hammerbacher wants to use his time on this earth to advance science, and Big Data is a means to that end.

You’ll get no argument from me here. It’s not about how much data you capture, store, or analyze. It’s about what you do with it.

Big Data: Big Results

From an article entitled “The Mayor’s Geek Squad” in today’s New York Times:

Now the city has brought this quantitative method to the exceedingly complicated machine that is New York. For the modest sum of $1 million, and at a moment when decreasing budgets have required increased efficiency, the in-house geek squad has over the last three years:

Leveraged the power of computers to double the city’s hit rate in finding stores selling bootleg cigarettes
Sped the removal of trees destroyed by Hurricane Sandy
Helped steer overburdened housing inspectors—working with more than 20,000 options—directly to lawbreaking buildings where catastrophic fires were likeliest to occur.

I love reading things like this. Is it still early in the Big Data game? Of course, but we’re seeing results every day.

Simon Says

As I write in Too Big to Ignore, regardless of your political affiliation, we all want–nay, need–government to do more with less. Recently, we have seen the rise of Big Data solutions, free public datasets, and an awareness of the power of Big Data. Against this backdrop, it’s time to start questioning many fundamental things–including the business of government.

Government can do more than less. Strike that. Government should do more than less. The default question should be: Can we use data to improve how we do things.

Feedback

What say you?

Google Reader and the Tension Between Users and Customers

So, Google is killing Reader, its RSS aggregator. Count me among those disappointed with the decision. As an avid Reader user, I’ll have to find an alternative–and relatively soon. (Netvibes seems to be the most promising so far.)

Note that, with Reader, I’m a user, not a customer. It’s a big difference that many of us often choose to disregard.

The Economist ran a thoughtful piece on the matter, citing how Google is providing a de facto utility for many of us–although the company is hardly a nonprofit. Perhaps Google will ultimately suffer for yanking popular products. Maybe in the future people will be less likely to use Google’s latest web services and apps for fear that they will one day go away.

That may very well be, but I certainly can’t blame a company for deciding not to support something that it essentially gave away. Truth be told, I would have paid a reasonable amount per year to use Reader. I doubt that I’m alone here. Google’s top brass ostensibly ran the numbers and decided that the costs of supporting Reader exceeded its benefits. Maybe Larry Page hopes expects people to get their news from Google Plus instead.

Whatever the reason, never confuse being a user with being a customer. The two could not be more different.