Daniel Miessler's Blog, page 87

The Rise of the Corporate Technology Ecosystem (CTE)

I love thinking about how technological advancement will intersect with our desires as humans.

My book on this topic is called The Real Internet of Things.

We have all these gadgets, technologies, and services, but towards what goal are they moving us? What’s the endgame? We see the features—like email and personal devices and various kinds of applications—but what’s the vision and story that brings it all together? That’s the piece that’s been missing.

I think the industry will soon figure out that the game is not producing a continuous supply of widgets and one-off services, but rather the holistic management of a customer’s life—centered around their core activities.

All the big tech companies will end up providing a complete, bundled Life Management System for their customers.

This isn’t the same as saying as a business—“Hey, we should really do something for people that like to exercise. Do we have any exercise intellectual property?” Or two quarters from then, saying, “Wow, we’re really getting killed in Education. We should have a learning app or something!”

Companies know they need to get into these various verticals, but most haven’t realized we’re ultimately talking about a Life Management System.

This type of thinking is still common because the tech industry is so new, and because most companies are so focused on next quarter’s (and next year’s) numbers. The piece that hasn’t happened yet is the realization that tech is for humans, and that the way to be successful long-term is to fully understand human life activities and workflows and then build a two, five, ten, and 20-year strategy around that.

Life as a collection of essential and valued activities

Humans change at the evolutionary timescale, so evolution is ultimately the customer.

The thing is, people are remarkably predictable. We have goals, and we have workflows. And our goals are determined by processes deep within us that change on an extremely long time-scale. This is something I talked about in the opening chapter of The Real Internet of Things.

Specifically, instead of starting with tech and seeing where it’s going, I’m starting with humans and what they seek, need, and desire. In other words, I think we can predict the future of technology through a strong understanding of what humans ultimately want as a species.

So if you want to know the shape of water—which can take any shape—your strongest play is to study the shape of the potholes (and other containers) it’ll end up in.

The Real Internet of Things, 2017

Humans are trying to win, becasue that’s what our DNA tells us to do. We want to be smart and attractive and productive and fulfilled and successful. We want to have a unique voice. We want to be creative. We want to be noticed.

And all of these are curation and optimization problems. You need to figure out the best foods to eat, when to eat them, how much of it to eat. You need to figure out the best way to spend your time. What’s the best show to watch? What’s the best way to learn how to make a new dish you had at a restaurant, or the best wine for a Game of Thrones watch party? Or the best gift for a friend?

Life is a constant stream of choices that we struggle to optimize to acheive our desired outcomes, and there is no higher calling for technology than helping us to manage those options holistically, across every dimension.

Another key for ecosystems will be knowing when to back off, because there is such a thing as too managed.

We’re talking about life optimization as a platform—from sleep, to eating, to learning, to working, to dating, to successful management of a family.

Ecosystem differntiation will offer lifestyle choice

This could also be called a Life Management System (LMS).

Choosing a Corporate Technology Ecosystem (CTE) will be much like choosing a religion, or a political party. There won’t be too many options, but the differences may be quite significant, and people will make their selection based on alignment with their life priorities.

Some of the key differences between ecosystems will be things like:

Aesthetic vs. utility
Functionality vs. anonymnity
Privacy vs. visibility
Personal assistant or life-line


Ecosystems will include safety, health, security, and privacy

CTEs will be holistic, meaning they will cover not just your work and leisure preferences and optimizations, but everyting to do with your security and safety. And that includes privacy.

Think about all the individual things you have to worry about today, as an individual, or especially as a parent. Health insurance, medicines, getting regular checkups, knowing quickly if someone gets hurt, making sure they get treated quickly, taking care of your parents as they age, dealing with legal matters.

Ecosystems will differ in what they emphasize and value

These are all risk related, and for the most part they’re one-off items handled by different organinizations or different groups. It’s many different conversations, with many different entities, to try to sort it all out. Well, that’s the point of an ecosystem.

The Soviet Union thought full life management should (and would) come from the government, but this type of holistic service is much better suited to corporations who benefit by having you using as many of their services as possible.

Managed privacy

DeleteMe is one such service that I’ve used in the past.

The privacy situation is especially interesting. Rather than ranting and chasing down various poeple who are stealing, selling, or otherwise misusing your and your family’s data, that will be included in these services. So instead of using some third party to go around deleting your data from places, or tracking down offenders, you’ll have a full corporate entity doing it on your behalf—using their giant team of privacy lawyers.

In order to get full use of the ecosystem, e.g., having doors open automatically, having your home automated, being able to buy things in a frictionless manner, etc.—this will require that your CTE has massive amounts of data about you. And that data will be collected in a continuous fashion.

Privacy doesn’t necessarily mean that nobody has your data, and in the future it’s going to mostly mean that only the right people have it.

But part of their service will be ensuring that only the right people have access to that data, in the right amount, and the right time. You’re simultaneously trusting your provider to have everything, but also enabling them to use all their resources to ensure that data doesn’t get out, and that it’s protected and used properly by everyone online.

So their job isn’t just using all of your data to make your life rich and productive—it’s also their job to make sure only that one ecosystem—along with vetted and trusted partners—can use it. And it’s up to them to keep it locked down.

The main ecosystem options

As with most such options—whether political or lifestyle in nature—the main two camps will be 1) curated, premium, and attractive…vs. 2) open, affordable, and ad-hoc—with one or two options that fall between or merge these in various ways.

The former types of ecosystem will be far more expensive, far more inclusive, far more premium, will be easier to use, will grant access to many higher-tier services, will invoke respect throughout the world, but will be more limited in your customization options.

The latter types of ecosystem will be far more affordable, open to virtually everyone, will be less polished, won’t have as powerful of branding or eliteness assocated with them, but will be far more open and customizable in terms of how you can mix and match smaller services into the ecosystem.

There will also be distinct strong points for various offerings—likely based on the origin story for that particular bundle. Some might be work-based, or activity-based, or lifestyle-based, for example.

A day in a life in an ecosystem

Let’s take a brief look at the life of someone named Amir in 2032. He lives in San Francisco and works at a design company near the Ferry Building. Amir uses the Oracle Ecosystem, because he loves saving more based on how much he does, and becasue they have his favorite mix of media and travel options.

His Life Management System bundle with Oracle Empower includes:

IoT Home (Full Digital Assistant access to his home and shared friends’ homes)
IoT Vehicle (Full DA access to personal and shared vehicles)
Personal Authentication to City and National IoT infrastructure
TaxID (Noah does his taxes for him, backed by Oracle)
VoteID (When he goes into a voting both, Noah validates his identity, and walks him through his options using an approved issues tutorial)



Oracle Health Gold (Full, 100% health coverage with very high limits and no deductables. Once he has a video visit with an AI Nurse or his chosen doctor, they give him a prescription that gets delivered within 30 minutes or that he can pick up at any Oracle Walgreens. Also includes full AI health monitoring and recommendations through Noah, which is his just a personalized Oracle DA, so if anything ever looks dangerous he’s instantly connected to a doctor with EMS standing by.)
Oracle Fitness (his 24/7 personal trainer with access to all the best Pelaton courses, personal instruction, and full Daemonized access to all Oracle 24/Hour Fitness Gyms Worldwide)
Oracle Sentinel (a Security and Privacy service that watches globally for signs that his data has been leaked or misused by any company, and responds with cleanups, cease and desists, lawsuits, etc. The system also constantly monitors the security settings for all his various digital and IoT assets.)
Oracle Scholar (Full digital access to the Top 350 worldwide universities’ courses, including live streaming events from the top professors and speakers in the world)





Amir wakes up to his alarm that selected the perfect time based on his sleep cycle.
His Digital Assistant (DA)—named Noah—shows him the weather, chance of rain, first meetings, and latest headlines on his full-wall display in his bedroom.
As he puts on his transparent glasses and watch, he gives his DA a full view of what he’s seeing.

The DA will recognize most every object and scene in Amir’s view using experience extraction.

As he eats his cereal, the calories are deducted from his total for the day based on the type of cereal he ate, how much, and the type of milk he used.
When he picks up his kettlebells, his DA asks him if he wants to do Kettlebells with someone else, and when he says yes it shows him a gym with an instructor doing a 10-minute kettlebell class.

His DA ordered the Uber through its API.

Read more about continuous authentication.

Upon finishing, his DA (Noah) reminds him, “The car will be ready downstairs in 6 minutes.” He gets in and gets out without fiddling with an app because Noah handles everything.
At work, everything works around him automatically, even things that require access or payment, because his DA stays authenticated to him continuously through various biometrics, so it’s doing things on his behalf.
While at work, he realizes he wants to send his girlfriend flowers, and he asks Noah to handle it. One of Noah’s jobs is to know the gift preferences of everyone Amir cares about, so he orders accordingly from api.flowers.com/order/, and then shows Amir what he ordered in his glasses.
When Amir takes a walk under the Bay Bridge after work, Noah chimes in and says, “You’re still favoring your right ankle. Do you want me to bring up Dr. Sarah on video?” Amir says yes, they have a brief video chat transparently in his glasses, and he pays nothing, and a girl on a one-wheeler hands him his prescription 7 minutes later while he’s under the bridge.
After his walk, Noah tells him that Steven Pinker is streaming a talk from Harvard in 3 minutes, and that he’ll background it in his glasses.

When he gets home his girlfriend is overjoyed because she just switched from the Oracle CTE to the one by Microsoft, and, “OMG, these features are so much better!!!”

When I watch something via any interface, it knows I’m an independent filmmaker, and it automatically imports the tools used by that director into my Creative Suite! Plus, I get a 2% Customized Meals discount when I make healthier choices, which are then recommended by my DA named Mindy!

Amir isn’t convinced yet, but it is nice to have the same CTE as the person you’re dating. He asks Noah to pull up the Microsoft Entourage teaser, which he does, but not without a slight delay and a bit of sadness in the voice.

Here you go, buddy.

Summary


We’re so early in the introduction of technology into our personal lives that most companies are still just throwing products as fast as possible into the market in hope that something works.
Soon a transition will be made that puts the core human activities at the center of technological innovation in the consumer space, which will open the doors to the Corporate Technological Ecosystem (CTE).
CTEs will include everything people do in life, from sleep to exercise to work to play to dating to travel to healthcare to financial and family management. Everything that singles and families already do—but curated and optimized 24/7—all for a single monthly subscription.

I could be wrong, but I think this is a matter of when–not if.

What kind of ecosystem will you choose?

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Unsupervised Learning: No. 164

This is a member-only even episode. Members get the newsletter every week, as well as access to all previous episodes, while free subscribers only get odd episodes every other week.

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

AI-powered Visual Sensors Will be Wielded By Digital Assistants to Build us Custom UIs

Benedict Evans is one of my favorite tech analysts. His content is usually great, I often agree with it, and his presentation style is quite unique. His annual presentation on technology trends should not be missed.

I had a chat with him in person once, and it was dissapointing to show him my long-term thoughts on where technology was going only to have him completely dismiss the ideas as obvious and vague.

Basically my book highlights what I think humans want from technology long-term, which can be predicted. And it then breaks down those desires into modules or components, which all fit together. I also talk about how some of the tech itself will likely work in a somewhat specific sense.

I wasn’t terribly offended about it, since he works for a a VC company that focuses on what they can invest in immediately. So the bias for the tangible was understandable.

Benedict didn’t really hear the specifics of what I was saying, but instead latched onto the grandiose, high-level of what I was saying. He heard, “And digital assistants will do this, and they’ll do that…”, and immediately assumed it was just capability prediction, and not anything useful in the short or mid-term. Crucially, he said everyone knew where we were going, and it was just a question of how we were goinn to get there.

I think this is quite wrong, actually, and over the last 5 years I’ve talked to a number of prominent technologists—including a few in charge of large research budgets—who, when they heard my entire outline, said that it was the clearest roadmap they’ve ever heard to what’s coming.

Anyway, this post isn’t about the meeting with Benedict, or how he is a bad person or anything. I think his position was completely defendable given his position and not having known me. And perhaps I could have presented it differently. Doesn’t matter.

What does matter is that I keep coming across more examples of my point.

In Benedict’s recent piece (which is excellent, by the way), Benedict writes about how cameras will be able to look at scenes and figure out more and more about what they’re seeing.

I’ve talked quite a lot about the impact of machine learning and computer vision in general on everything from e-commerce recommendation to social to all kinds of cool industrial applications, but it’s also interesting just to look at the effect that machine learning is having on actual cameras. 

Source: Cameras that understand: portrait mode and Google Lens — Benedict Evans

He goes on to talk about how this will allow AI to answer deeper and deeper questions.

So, we are clearly well on the way to the point at which any photograph a normal consumer takes will be technically perfect. However, there’s a second step here – not just “what is this picture and how should we focus it?” but “why did you take the picture?”.

Source: Cameras that understand: portrait mode and Google Lens — Benedict Evans

This is precisely what I talk about in the chapters, “Algorithmic Experience Extraction” and “Businesses as Daemons”, where the feed from your sensors on your body will be shipped to dozens or thousands of very specific AI algorithms that will completely decipher what is being seen.

Humans don’t care about video feeds. What they care about are events. We want to see first kisses, love triangle fights, humans reacting to new media, car crashes, rescues, heroism, cowardice, and everything in between. We want to see life, and that’s precisely what the algorithms will provide us.

Source: Miessler, Daniel. The Real Internet of Things (p. 17). Kindle Edition.

The point of this is not to show that Benedict was wrong, or I was right—we’re both right, and we’ll both be wrong about some part of this as it moves forward.

I believe this enough to have three patents in-flight around what I talk about in the book.

The point is that I still believe it possible to dive deeper into our human requirements on technology, and to effectively and usefully reverse engineer specific technologies that can make those things happen.

The core premise of my book is:

The human has desires and needs, some passive and invisible (feeling depressed), and others are explicit and overt (clicking a button or giving a command).
The DA will be consuming all these pieces constantly.
The DA will also be parsing everything in the real world constantly, extracting meaning, and matching it to the principal’s state.
This way, the DA can anticipate requests, do things in the background, and generally improve the life of the principal in a continuous way.

Examples here include opening doors for you as you walk, having your food ready for you when you get into a restaurant, having your favorite sport playing on the display when you get there, etc.

Benedict is rightly talking about how computers will interpret inputs using language translation, and object identification, and this will all happen at increasing levels of complexity.

But who’s being obvious now? Of course that will happen.

The real question is how all those things will combine to fulfill real human needs and desires, and that is what I’m interested in. That is what I wrote my book about.

It is useful to move as far forward as possible into the tech, because you can then move backwards looking for milestones that we can possibly hit to make the longer term picture possible.

But we can’t do that if everyone is thinking short-term, which I think too many are.

Benedict thinks everyone sees what I see, but I think he got that wrong becaue he didn’t see it either.

The key innovation here is not the AI interpretation of objects in the view of its sensors. The key innovation will be the DA that merges those interpretations and matches them to current states within their principal, and then uses that combination to make requests to external Daemons (which are Businesses as Algorithms) to make things happen.

I understand that this is further out, but it doesn’t make it any less inevitable than the evolution that Benedict is talking about. It’s not like this requires AGI to happen. These are small incremental steps adding to each other until something extraordinary happens.

And my only point—in all of this—is to say that anyone trying to build the whole thing needs to keep their eye on the bigger picture as they build their incremental steps.

The companies that do this best will have a major advantage.

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Lifestyle

This page is where I collect all my lifestyle-oriented items, meaning material things I enjoy collecting or enjoying in some way. These are things like, watches, cars, beverages, foods, travel locations, restaurants, etc.

The higher version of myself sees this all as terribly shallow and empty, but I think it’s ok to indulge in this mode of living for short periods of time. The trick is to not let it consume you.

My primers on various topics

First, I have a collection of basic guides to topics. I wrote these so that I could appreciate the subjects more thoroughly when partaking, and also to not look like a Neandertal when confonted with them in social settings.

Wine
Beer
Steak
Whiskey


My tastes

For watches, I have a Rolex Submariner, which is the watch I’ve wanted since my early 20s and finally got as a gift in my 30’s. I now mostly wear my Apple Watch, however, and I doubt I’ll wear another traditional watch until they merge smart capabilities. The next watch I would have purchased would have been the NOMOS Tangente.
For cars, I have been an exclusive BMW owner for over 15 years.

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Whiskey

I never learned to drink until I was in my 30’s. I always thought alcohol tasted horrible, and am basically immune to peer pressure, so the combination of those two meant I could have gone my whole life without it.

The friendship thing is especially true for introverts like myself.

But then I realized that drinking can be about people and friendships—and that it has layers and levels to it just like anything that’s popular over thousands of years. Ultimately, if you enjoy things like watches and cameras, it’s another lifestyle item to deep-dive on just for the fun of it.

So here’s my basic primer on the topic, including how it relates to Scotch.

Basics

The word whiskey comes from the gaelic word for water (uisce), and in Latin it was known as aqua vitae (water of life).

Whisky is a distilled beverage from fermented grain mash.
Grains are the dry fruits of the plants in the grass family (Poaceae), and contain starches that provide their calories.
A cereal is the edible components of a grain.
Barley is a type of grain, and it’s the fourth largest type of crop behind wheat, rice, and corn.
Barley can be malted, which means wetting cereal grain with water until it germinates, and then drying it with hot air. This develops the grain so that its starches can be turned into various sugars.
Many grains can be used for different types of whiskey, including barley, corn, rye, and wheat.
You typically age whiskey in wooden casks, which are usually made of white oak.
The variations of whiskey deal with the grains used, distillation processes, and aging in wooden barrels.

Barley’s botanical name is Hordeum Vulgare, which I like to read as Vulgar Horde.

Distillation appears to have come from the Babylonians, to the Greeks, to the Arabs, and then to Italy.
Distillation of alcohol appears to go back to Italy in the 13th century, where they used wine (of course).
Distillation of alcohol came to Ireland and Scotland in the 14th or 15th century.


Types of whisky

In the U.S. and Ireland it’s spelled Whiskey, and in most of the rest of the world they leave off the “e”: Whisky.

Malt whiskey comes from malted barley
Grain whisky is any kind of whisky that does not come from malted barley—although keep in mind that barley is a grain. Basically, barley is like the elite grain, and anything else is called “grain” while it’s called barley.
Blended whisky can contain any combination of whiskys, including malt and grain.
Single pot whiskey is from Ireland and is distilled only in pot stills.
Bourbon is from anywhere in the U.S., but mostly from Kentucky.

Whiskey is basically beer without distillation or hops (the flower of the hop plant).

Bourbon must be at least 51% corn, combined with a mix of barley and rye and/or wheat.
Bourbon requires an age of at least 4 years in American white oak.
If the Bourbon is not blended it can be called “Straight Bourbon”.
Rye whiskey is at least 51% rye, and it’s fallen out of favor due to its spicy taste.
Corn whiskey is mostly from the U.S., and must be 100% corn to earn the name.


My preferences

First, I’m a novice, so don’t take this too seriously. Plus it’s all highly subjective. I myself prefer smooth, subtle, and nuanced beverages—whether it’s coffee (Aeropress Philz Medium blend), Beer (Sam Adams, Asahi), or wine (Pinot Noir, Sirah).

My favorite wisky right now is actually Japanese.

I doubt I’d even be able to differentiate well between similar options in a blind taste test becasue I’m such an amateur, although I do really notice the smoothness of the Japanese options.

Yamazaki 12 Year
Hakushu 12 Year
Johnny Walker Black (I have a bias becasue it was Christopher Hitchens’ favorite drink)
Glenfiddich 30


Notes


I’m obviously not saying that drinking is without its own risks, so I am definitely not giving some open endorsement.
Some of this content is right from the Wikipedia article; I’m just curating from multiple sources into the basics. Link

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Technology

I’m in cybersecurity, so when I write about specific technical fields, I try to do so with the humility of a layperson.

I write a lot about technology because I think it’s profoundly important. Not for its own sake, but because it affects the trajectory of humans progress.

My stated purpose here is to:

Explore the interesting and beautiful at the intersection of security, technology, and humans.

…and this really captures it. When I write about technology I prefer to come from the perspective of how it affects people. But I also do provide more technical resources for those who need something specific.

Below are a number of my essays and tutorials on multiple technical topics going back to 1999.

My Technology Blog Posts

The future of technology and humans


Machine Learning Will Magnify Humanity’s Strengths And Weaknesses
What is Technology Racing Towards?
The Biggest Advantage in Machine Learning Will Come from Superior Coverage, Not Superior Analysis
Personal Sensors Are the Next Breakthrough in Consumer Electronics
My Current Predictions for Thinking Machines (2017)

Full Tutorial List

My technical tutorials


A vim Primer
A Tcpdump Tutorial
Character Mapping and Encoding
Big-O Notation Explained
THe Difference Between URLs and URIs
Website Performance Primer

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Happiness

By Kalen Emsley

I’m in cybersecurity, so when I write about topics like health and psychology, I try to do so with the humility of a layperson.

I write a lot about happiness—not because I’m an expert on the topic but because I think it’s a topic that matters to everyone.

I see a robust pursuit of happiness as dependent on a deep understanding the world. It touches so many different fields—from psychology, to spirituality, to economics, to politics—which is perhaps why it’s hard to find true experts.

The essays below outline how I’ve approached this journey over the years, and I hope you find some part of that to be useful.

Evolution, struggle, and meaning

Information Security

Information Security—also called Cybersecurity—is arguably the most interesting profession on the planet. It requires some combination of the attacker mentality, a defensive mindset, and the ability to constantly adapt to change. This is why it commands some of the highest salaries in the world.

InfoSec is such a big field, however, that it’s useful to break it up into sections. I’ve done this over the 20 years that I’ve been writing about security, and here are some of the topics you might find interesting.

Sales and marketing teams often conflate these definitions, leading to confusion in the industry.

Offensive Testing: When to Use Different Types of Security Assessments, The Difference Between Pentesting and Red Teaming, The Difference Between Threats, Threat Actors, Vulnerabilities, and Risks, The Difference Between Events, Alerts, and Incidents, Security Assessment Types





Security Tools: Shodan, Masscan, Nmap, Tcpdump, Lsof, iptables

My cybersecurity career guide takes you step by step through the process of building a security career.

Building a Security Career: Building a Career in Cybersecurity, Information Security Interview Questions, Cybersecurity Lacks Entry-level Positions





Security Philosophy: Secrecy is a Valid Security Layer





Security Concepts: Encryption vs. Encoding vs. Hashing, Information Security Definitions




Attack


Security Assessment Types

The Difference Between a Vulnerability Assessment and a Penetration Test

The Difference Between Red, Blue, and Purple Teams

A Masscan Tutorial

A Bettercap Tutorial

How to Use Shodan

When to Use Vulnerability Assessments, Pentesting, Red Team Assessments, and Bug Bounties

Purple Team Pentests Mean You’re Failing at Red and Blue

An nmap Primer




Defense


Obscurity is a Valid Security Layer

An iptables Primer

The Difference Between Events, Alerts, and Incidents

Information Security Metrics

Same Origin Policy Explained

Serialization Bugs Explained

A Security-focused HTTP Primer

Vulnerability Database Resources




Assorted


My Information Security Blog Posts

Information Security Definitions

The Difference Between Threats, Vulnerabilities, and Risks

How to Build a Successful Information Security Career

The Birthday Attack

Information Security Interview Questions

Encoding vs. Encryption vs. Hashing

Diffie-Hellman Explained

The Difference Between the Internet, the Deep Web, and the Dark Web

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Unsupervised Learning: No. 163

Unsupervised Learning is my weekly show where I spend 5-20 hours finding the most interesting stories in security, technology, and humans, which I then curate into a 30-minute podcast & companion newsletter.

The goal is to catch you up on current events, show you the best content from around the web, and hopefully give you something to think about as well.

Subscribe to the Newsletter or Podcast

Big-O Notation Explained

I remember hating my algorithms class in college, and Big-O notation was part of the reason. I just never got it. It seemed completely disjointed from anything I could possibly do in real life. If that’s you as well, here’s an explanation that should clear it up for you.

How much harder does something get as you increase your input?

An algorithm is just a series of steps for solving a problem.

Big-O notation is a metric for algorithm scalability.

The entire point of Big-O notation is to be able to compare how efficiently one algorithm solves big problems compared to another.

So if you have a giant list of names, or phone numbers, or some other kind of input—what’s the best possible way of sorting them, or searching them, based on whether the list has 10 entries, or 100, or a million?

That’s what Big-O notation tells you. It tells you how well that particular approach will do against very large problems.

O(1)


/’oh won’/


constant complexity
No matter what you provide as input to the algorithm, it’ll still run in the same amount of time.



1 item, 1 second
10 items, 1 second
100 items, 1 secon

Example: Determining if a binary number is even or odd.

It’s helpful to think of logarithms as exponents in reverse.

O(log n)


/’oh log en’/


logarithmic complexity
The calculation time barely increases as you exponentially increase the input numbers.



1 item, 1 second
10 items, 2 seconds
100 items, 3 seconds

Example: Binary search.

O(n)


/’oh en’/


linear complexity
The calculation time increases at the same pace as the input.



1 item, 1 second
10 items, 10 seconds
100 items, 100 seconds

Example: Unsorted list search.

Things raised to the second power are called quadratic because it comes from the Latin for “making square”, i.e., to multiply times itself.

O(n2)


/’oh en squared’/


quadratic complexity
The calculation time increases at the pace of n2.



1 item, 1 second
10 items, 100 seconds
100 items, 10,000 seconds

Example: Bubble sort.

N! is a bad place in the algorithm world. It means it’s basically unsolvable.

O(n!)


/’oh en factorial’/


factorial complexity
The calculation time increases at the pace of n!, which means if n is 5, it’s 5x4x3x2x1, or 120. This isn’t so bad at low values of n, but it quickly becomes impossible.



N=1, 1 option
N=10, 3,628,800 options
N=100, 9.332621544×10157 options

Example: Traveling salesman.

Summary


Algorithms are lists of steps for solving problems.
Some algorithms are good at problems when they’re small, but fail at scale, e.g., with very long lists of things to sort or search.
Big-O notation is the way to tell how good a given algorithm is at solving very large problems.


Notes


There are other Big-O notation types as well, but these are the major ones. Link

—

Subscribe for one coffee a month ($5) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month.