Daniel Miessler's Blog, page 106

The Real Internet of Things: Summary of Concepts

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: Summary of Concepts.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

We covered a lot of ideas here, so let’s just review the main points.

There are basic trends in technology that we can see crystallizing: centralized to peer-to-peer, forced to natural, obvious to invisible, manual to automatic, periodic to continuous, scheduled to realtime, private to open, visual to multi-sensory, aggregated to curated, and designed to evolved.

Objects, including humans, will become their own authoritative sources of truth for information about them through a concept called Universal Daemonization.

Universal Daemonization will allow for realtime data gathering about objects in the world at nearly any scale.

Daemonization will be bi-directional, allowing for the updating of, pushing to, and issuing of commands to other objects.

Because humans will not be able to parse and interact with billions of daemons, Digital Assistants will perform this task for them.

Digital Assistants or Advocates (DAs) will work to optimize the life of their principals continuously, without rest, 24/7/365, and in multiple threads.

Our DAs will use AI to subtly alter their principals’ interface to the world, giving them better knowledge and providing interfaces for modification.

All requests made from your daemon will be made as a centralized identity, and third parties will validate that it was truly you that made those requests.

Our daemons will display numerous reputation scores about us which are also validated by third parties.

Our DAs will constantly customize our experiences around us by modifying what things look like, how they’re configured, and how we experience them, through transparent daemon interaction.

Constantly improving algorithms will be connected to the billions (then trillions) of sensors in the world, and they will constantly parse reality into events and data that are meaningful to humans that can then be shared with various entities.

Our DAs will use whatever resources they have (thousands of eyes and ears) to monitor us, our loved ones, and our valuables for safety and security.

Our DAs will functionally grant us superpowers through the enhancement of our sight, hearing, and many other sense types that we don’t even naturally possess.

Businesses will become daemonized forms of their core algorithms, and the primary consumers of these business daemons will not be humans themselves, but rather their DAs using the services on behalf of their principles.

People will broadcast their third-party-validated capabilities through their daemons, and this peer-to-peer infrastructure will represent the future of finding and securing work.

The four ways of gathering and using data will be: realtime data from objects, transferring that data, analyzing the data using algorithms, and then presenting it in some useful way.

The combination of machine learning and evolutionary algorithms will not only improve our ability to learn about the world, but will improve our ability to improve that ability.

This will culminate in a framework that allows humankind to systematically define its goals, study reality in realtime using AI, and then make optimizations to our behavior that best lead to our desired outcomes.

Daemonization will ultimately allow humans to reduce their reliance on large, abstracted institutions and instead look to each other for their needs.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

The Real Internet of Things: Details and Examples

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: Details and Examples.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

Each of the chapters you’ve read so far have introduced a single concept per section. I did this to make the concept crisp and simple, which isn’t possible if you start talking about how it might be implemented.

In this section I give a few additional ideas and likely/possible applications for each.

Universal Daemonization

Most objects will have an /info endpoint in their daemon that allows other objects to understand its basic attributes. This might be where the schema is stored, i.e., all the different endpoints that are available (that can be seen by the current viewer), etc.
Restaurants will have /menu and /order and /payment endpoints. Buildings will have /safety, /blueprints, and hundreds of other /infrastructure endpoints, and these will become quite standardized over time. The same will apply to millions of obvious entries for certain objects. Books will have info on the author, the length, where they were written and when, what tools were used, the list of references, and countless other pieces of information that you’d expect to see. The key is that these will become standardized for the basic info types for each object based on the object type, even though they may have many custom endpoints as well.
Consumables and construction materials will have, as part of their daemons, a series of elements related to lifespan, integrity, etc. These values are updated through numerous means: time in use, official inspections, etc. So a city can simply look at 10 city blocks and instantly evaluate what structures are most in need of upgrades. And consumers’ DAs can read a principal’s entire list of things they own, and order/recommend replacements based on it being time. Car tires, bike tires, the roof on the house, toothpaste, polo shirts, chicken soup, diapers, etc. The key point here is not just that someone (like the manufacturer) knows these things and can nag you—it’s that the object itself will know, and that your DA can manage replacement according to a combination of requirements and your preferences.
The more up-to-date an object’s information the better, so the informational endpoints will have update/validate options available for external clients that aren’t able to make changes to the daemon itself. For example, if a mobile business is marked as being at a certain street corner, but it moves without updating itself for whatever reason, it will be able to accept update requests from people who see that it’s moved in the last minute. This will depend on the authority, reputation, trustworthiness, and how many votes come in that agree. Entire economies will likely rise on paying micro-payments for providing high-quality validation of reality. The status of businesses, the location of common objects, ratings of service, etc. People could possibly make a living by simply observing and reporting in a responsible and valuable way, with updates being pushed right into the objects’ daemons themselves. – Even basic information about an object will have their own ratings based on a number of these factors—how often they’re updated (per day, per minute, per second, etc.). How many external validations have been made with what authority, etc.
Daemonization won’t be just for physical objects like vehicles and buildings and people. They will also be used for (and useful to) conceptual and virtual objects such as applications, systems, etc. Think of the use case of an IT asset database, where all applications, servers, operating systems, tests, builds, vulnerabilities, etc—all have their own daemons and their own realtime status. This makes queries and updates simple and elegant—exactly as they should be. When you run a security scan and find vulnerabilities, they each have their own daemon with their own schema, and they’re attached to an application that has its own, which sits on top of an operating system, which sits on a piece of hardware, etc. So finding out what version of software sits on what OS, what data is being used where, how that data is changing, where it’s moving—these all become daemon status updates.

Realtime Data

One of the biggest advantages of realtime data will be serving as a constant stream into scientific studies. So rather than data analysis taking days, weeks, months, or years, we’ll be able to pull that data ever day, every hour, every minute, or multiple times per second, constantly, and at scale.
Realtime data also allows us to more quickly study the effect of variables in these scientific studies. If we’re able to track mood in realtime, for an entire city, then we’re more likely to be able to say that a particular variable was the cause of change in that mood.
Realtime data will also be able to speak to, at a more granular level, the difference between causation and correlation.
Ultimately realtime data is the most important component in the information infrastructure and Desired Outcome Management concepts, since it’s the data that’s feeding the algorithms and output.
People or systems can simply ask questions about the state of the world and get answers, e.g., how many dolphins are there within one mile? is this area more star trek or star wars? How many airplanes are there over my head, which country’s currency has lost the most value in the last hour, how many single people who prefer cats to dogs are within a 10 minute drive, what’s the most popular fast food in this area? These types of queries will obviously require the other pieces of the information infrastructure as well (data transfer, algorithms, etc.), but the data is the most important piece.

Digital Assistants

DAs will basically run our lives, from wakeup time (based on your sleep cycles and the latest research), to which method works best (raising the lights and playing certain music), to starting your favorite food and beverage, preparing to read/display your preferred news sources, etc.
Your household, work environment, and any other place you spend time in will be run by your DA as the custodian of your life ecosystem (you’re the owner). When you buy new equipment it will of course be daemonized, and the enrollment process will involve it being added to your digital ecosystem. That means it’ll be automatically hardened and access controlled based on your ecosystem. So if friends and family can do certain things with certain kinds of devices, but not with others, those settings will automatically apply to this system as well.
Instead of household items like food and dish soap and paper towels ordering replacements for themselves, i.e. talking directly to businesses, every household item will register with the head-of-household’s DA, and the DA will manage the household based on its knowledge of preferences, calendars, etc.
Your DA will notify you in certain ways, which will over time produce Pavlovian responses, when certain situations arise. You’ll hear a sound when a single person of the opposite sex nears you while you’re not working, but only if they pass a few filters that are important to you.
You might let your DA use a number of commercial algorithms to find matches for you that you wouldn’t have thought to explore yourself. So you may put yourself in Cupid mode, or Spontaneity mode, where two DAs create pre-filtered but semi-chance meetings between two principals.

Tireless Advocate

If someone mentions to you casually about a particular sport, your DA (knowing you like to immerse yourself in new hobbies) will find the nearest training locations, the best local trainers, the best and nearest places to play, and some top tips for getting into shape. So when you inevitably ask about it in the next day or so, your DA will have an entire plan sorted out for you.
Any research topic you express interest in, or ask your DA to look into, will get a full parsing and summary treatment, ultimately resulting in a summary (which probably comes from its own commercial API) that gives you exactly how much information you wanted on that topic.
Summaries will have depth levels, so you’ll be able to say things like, “less depth”, or, “more depth” as desired, but that’ll only be when it doesn’t get it right in the first place.
DAs will scour the world looking for negative information about you, news that could negatively affect you, etc., and will bring it to your attention if it finds something.

Augmented Reality

When speaking to someone either in person or remotely, you will see indicators in your field of vision telling you how truthful they’re being. This will be from voice analysis, facial expressions (if it’s a visual call), etc. The visual indicators might be a Pinocchio nose, a red outline around them or your field of view, or it may be a non-visual indicator, like a subtle hissing or vibration.
As you’re talking to people you’ll have metadata about them displayed, such as humor scores, attractiveness ratings, favorite foods, favorite books, and interesting connections to you like mutual connections, that they attended the same college, etc.
Single people in public places will automatically see potential matches in different ways, e.g., displaying a cupid over their heads, showing them in color while everyone else is grayed out, etc.
The context that you’re currently in will be explicitly set by you or automatically set by your DA. If you haven’t eaten and your stomach rumbles your DA might switch you into “food finding mode”, which displays restaurants near you in particular ways based on what you’ve eaten recently, your favorite type of food, which places have the best ratings, etc.
Kind people will be able to turn on the Gloomy filter and see a gray and raining cloud over the heads of people who need cheering up.
You’ll be able to see live crime statistics for the area you’re currently in whenever it’s after a certain hour and you’re in an unfamiliar place.
Companies will specialize in providing artfully subtle yet powerful AR indicators for various contexts, e.g., hungry, lustful, angry, skeptical, curious, tired, frightened, sad, depressed, euphoric. Each of these will have different displays in your visual field, ambient and directional sounds, subtle vibrations in parts of your body, smells, etc.
Becoming frightened in a strange area could outline everyone in green or red to indicate who to avoid or seek help from based on facial imagery, gait analysis, body language, etc.—all of which is being streamed in realtime to a series of business daemons that specialize in this type of analysis and UI/UX display.
When people are sad or angry, your DA will stream the situation and the context to a company/algorithm which will display the perfect thing to say to solve the problem. It could be a de-escalation phrase, or a phrase that takes responsibility, or shows empathy, or whatever that situation needs.

Identity and Authentication

As you move throughout the environment, whether at home or overseas, doors will open or not open based on who you are and what level of authority you have according to that resource. If you’re a police officer in the United States, for example, and you are in Munich, you might be granted access to POV access to a certain street camera, while your friend who is not in law enforcement will not.
When you purchase (or lease) a new object, such as furniture or technology, you’ll simply enroll it through your DA into your ecosystem. All of your preferences will be automatically applied to it, including how it’s locked down, who can access it, under what circumstances, etc.
When you sign up for new services and experiences, your DA will transparently convey both your authentication validation (which will be signed by your Identity Verification Service) and all your preferences.

Reputation as Infrastructure

You ask your DA where to go for a weekend trip, and it calculates all the variables based on the best experience, price, and ratings by people in your network who have gone there. Your DA recommends the winner and then uses a separate daemon/business/API to build the travel plan and add it to the calendar
There will be algorithms/companies that do nothing but find special combinations of high ratings in random things. Like making people laugh, combined with being able to bake, combined with cosmology knowledge, and will use this cocktail to recommend relationships or problem-solving connections for people.
People rated extremely high in altruism and selflessness in terms of actually giving time and resources will be able to soak up micropayments from those around them. I may have a small amount of money that I transparently give to those like that around me, for example, and they might be able to just go through life giving of themselves without worry of where to eat or sleep. It’s an extreme case but it will be possible, especially with local businesses helping with free goods.
People will be more likely to treat others well since they won’t want their selfishness ratings to rise too high, which could lead to paying higher prices for things, not getting access to certain places, or people choosing not to interact with them.

Continuous Customization

Walking into a sports bar could see the content on the displays change, the music over the speakers change, etc. You could get one waiter vs. another, be asked about your day or not, have the temperature in the place raised or lowered, etc.
If you work at a physical location, your settings will instantly transfer, including your desk height and angle, your chair settings, the lighting in your cube, your communication settings, how often you are to be interrupted, etc.
When you visit a hotel your DA will have everything configured for you according to the maximum capabilities of the property. This will include bed style, products in the bathroom, what’s playing on the display, the temperature in the room, etc. These are not things that you ask for—they’re all things that your DA knows best about you, and it simply transfers that to the property as a customization package request.
As you move from place to place (say hotels or airplanes) your context will transfer with you through your DA. If you’re halfway through watching a show on a plane when you land, your DA will ask if you want to pick it up where you left off when you get in bed at the hotel.
Anything that’s customizable that you visit will be adjusted by your DA in this way. Walk into a new home dealership and you’ll hear your favorite music, or an ideal experience for that environment, you’ll get your favorite drink, and someone will interact with you in a way you’ll enjoy.
Customization will include spontaneity, since your previous likes will inevitably get old. Part of your DA’s job, as well as the job of various experiences, will be to delight you with new music, new lighting, new interactions, etc. This will be handled through thousands of competing businesses/algorithms/daemons.
At certain really important times in your life, like when you’re out riding bikes with a childhood friend in your hometown, you might both hear a perfect soundtrack of the music you used to listen to together. These are perfectly curated experiences, performed by your DA, using specialized experience companies/algorithms. The purpose of these APIs is to always have the perfect song, or the perfect view, or the perfect whatever, for that situation.

Omniscient Defender

You’ll be notified by your DA if anyone in your family is in a dangerous situation that falls above a certain threshold.
You’ll be able to switch your visual point of view instantly to any camera you have access to, whether that’s inside your house, through the eyes of someone you’re sharing access with, drones hovering over your house, etc.
You and your loved ones’ DAs will be scanning the environment for situations that seem dangerous. More specifically, they’ll be streaming live footage to their preferred company/algorithm that specializes in assigning danger ratings to environments.
Your DA will stream surrounding conversations to a company/algorithm to alert you of anything that could be dangerous to you.
Your entire ecosystem becomes eyes, ears, and noses for your DA to monitor your family and possessions 24/7.
Your DA will let you know when you are being monitored, when you should disable your daemon, take other countermeasures, etc.
Your DA will let you know when you’re doing something that could be controversial, and give you the capture points that it could be pulled from. It will be constantly monitoring ways you could be monitored and make sure that you’re not causing harm to your reputation.

Human Enhancement

As you move into new areas your DA will query and load up various views of different objects you might look at. So you could look at a building and see the people inside as heat signatures, or you might see the schematics of a building from a distance, all as AR overlays.
With a gesture or a voice command you will simply enable X-Ray vision, or heat signature, and then see the world through that perspective. Things that were pre-loaded will show quickly, others will take time, and others will not show at all because there’s no data for them. But over time, and with enough daemon access, we’ll be able to see much of the world in this way.
The same will apply to hearing, with the ability to focus on someone across a parking lot and hear what they’re saying. You’ll be able to gesture, target, or do some other aesthetically appealing way to initiate and visualize the effect. When it kicks in you’ll hear the person talking as if they’re speaking in your ear.
You’ll be able to zoom into things visually as well, and things you can do this with will be subtly indicated within your AR interface, perhaps with a tiny telescope in the top right corner. For any of those items you can make a gesture and zoom in and out using whatever nearby cameras that exist (that you have access to).

Businesses as Daemons

Play the perfect song for this moment.
Find the perfect gift for this person.
Write the perfect letter for this situation.
Create the perfect song for this situation.
Give me the best path from A to Z given this cargo and time of travel.
Display this content to me in the way that will help me make the best decisions in the least amount of time.
What should I watch right now?
When I look at the cover of a book, give me a perfect summary that fills the cover, along with the rating.
What should I listen to?
Why am I not happy?
If I redesigned my living room, what are the top best options?
What do I waste the most time on in my life?
Surprise me with an interesting music choice that I’ll love but never would have picked myself.
Remind me when there’s someone I care about that I haven’t told how much I love them.
Show me how much danger I am in at any given moment.
Show me the local crime statistics for the area I’m in.
Spend up to 5,000 to automatically deploy local defenders if anyone in my family gets in danger.
Only show me menu items that I should eat as part of my new health plan.
Build me a perfect daily routine based on my life goals.
Find the perfect girl for me.
What comic series would I like?
I’m new to Sushi, what should I try on this menu?
I just got this text, how should I respond?
I need to impress this person I just met; build me a weekend spending no more than 1,000 that they’ll love.
Who’s hurting the most within 1 mile, and what can I do for them?

The Future of Work

Jason is rated highly in many local and global skills, and he sits relaxing at his favorite coffee shop. He’s asked his DA to only bother him with incoming job requests if they pay over a certain amount. Because of his high ratings in these skills, he often gets fiction editing requests, requests to help move things, cat-sitting, legal contract review, and empathic listening. When a job passes the threshold, his DA (named Timmothy), will break in quietly in his earpiece. “Legal contract review, 37 pages, due by tomorrow morning, are we interested?” Jason nods his head and the details are worked out between DAs transparently.
Nadia sits at her favorite co-working location watching incoming job requests scroll by her AR display. She’s a coder so she’s watching the logos for various languages scroll by, with the size of the icon representing the payment for the project. She also gets lots of requests for helping people with programming, which she sometimes takes just to be nice, since they don’t pay much. She sees a big project in her favorite language scroll by and tells her DA (Vira), “I’ll take that one.”
Companies will specialize in finding better matches of job seeker and job taker based on hidden truths they extract using their proprietary algorithms.
You’ll be able to simply say, “Find me someone to help me move this pile of rocks,” or, “Find me the best person to edit this photo.” Your DA will contact multiple companies/algorithms to find the best fit for you (if it doesn’t know already) and that company will connect you with the best service or person.
You’ll be able to specify that you prefer local, that you prefer in person, that you prefer best in the world, that you prefer cheap, etc. And the algorithms your DA uses will take this into account.

The Four Components of Information Infrastructure

There will be countless companies competing to provide various parts of the sensor, daemon, transfer, algorithm, and interface infrastructure. They will be modular in nature so that one company’s sensors can interface cleanly with another company’s daemon technology.
Each piece is a bottleneck to the entire system, so realtime data for increasingly large systems (consisting of billions and trillions of nodes) will become possible as the minimum capacity component in the infrastructure is upgraded at any given time.

Getting Better at Getting Better

Machine learning and evolutionary algorithms will be used to improve each other, accelerating the pace at which both can improve.
As algorithms start replacing humans, the pace of improvement will only increase; shown by algorithms getting better at handling customer service issues, driving, obtaining knowledge, etc. The advantages of these gains can be almost instantly transferred and applied elsewhere in the industry.

Desired Outcome Management

Algorithms will be pointed to realtime datasets that are continuously updated, and the algorithms will continue to optimize themselves as they consume more data.
Philosophy will become far more interesting because once the data infrastructure is more standardized and modular, the value from data, statistics, and machine learning will increasingly hinge upon asking the right questions. The right questions, in turn, will hinge upon our goals and will unfortunately require philosophers to figure out (and articulate) what those should be.

Peer-to-peer Value Exchange

Eventually people will build communities that are functionally, emotionally, and economically tied to each other through mutual dependence and support. Value exchange, providing of services, infrastructure mechanisms, etc.—will all be handled internally to a significant degree in order to return to a sense of community.
Fewer services (but still many) will require centralized, institutionalized resources, as the peer structure will eventually be robust enough to handle most needs. Exceptions to this norm will be in the areas where it makes sense, construction and emergency vehicles for example, to have a permanent staff available at all times.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

The Real Internet of Things: Peer-to-peer Value Exchange

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: Peer-to-peer Value Exchange.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

We’ve talked about how the future of work is largely person-to-person interaction mediated by a daemon-powered tech layer, but the peer-to-peer model goes far beyond employment.

What daemonized peer-to-peer really enables is less reliance on centralized institutions.

If you are in need of medical attention and there are 38,761 people within one square kilometer, it may not make sense to call on a centralized authority to provide that service. What if, upon injuring your leg in an accident, your DA could simply beacon out to nearby people. Less than 90 seconds later someone with the proper training, equipment, credentials, and ratings shows up and provides assistance. A micropayment of currency, appreciation, and a high rating is sent from daemon to daemon and the two people go on their way.

The same will apply to safety. Imagine a woman walking alone in a dangerous area and receiving a notification from her DA:

It’s not safe here. I’m getting you some company.

Within a few seconds she’s joined by three other people on the street (outlined in green within her view) who smile and walk with her to her destination. There is another exchange of appreciation, smiles, and/or currency, which is reflected on both sides.

Now think of how this could apply to building homes, providing fresh and healthy food, and many other core human needs. Institutions will still have a role, of course, but we the people are in fact the ultimate institution.

Daemonization will allow us to provide ourselves with what in the past needed to be abstracted. It’s bottom-up vs. top-down at the ultimate scale.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

The Real Internet of Things: Desired Outcome Management

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: Desired Outcome Management.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

Now that we’ve talked about the infrastructure for collecting, analyzing, and presenting information, we can move on to a concept I call Desired Outcome Management (DOM).

The assumption underpinning DOM is the simple claim that we want to improve things but we don’t know exactly how to go about it.

DOM provides a model for improving almost anything, and data plays one of the central roles.

DOM is broken into a few main components:

Define your goals. This could be for a business, a city, a family, a department, a country, a team, or an individual. Examples are things like: graduate from a top-10 university, make 100K/year, reach the top 10 ranking in quality of life, attain 150K in passive income, have a happy and fulfilled family, etc.

Define your model. A model in this case is a method or approach for attaining a goal or set of goals. For example, if you want to live a fulfilled life, there might be a Tony Robbins model, or a Dr. Phil model, or a model you make for yourself. It’ll have statements in it like, “You need to be healthy to be happy. You need to exercise. You need to eat plenty of raw foods, etc.”

Capture data. From there, you need to capture data about your entity’s behavior, from the real world, and get it into the system. So if you have a model that talks about diet, you need inputs regarding what you eat, how much you exercise, etc. If your model cares about grades in school, you need to get those grades into the system.

Provide Ratings. Next your system needs to provide clear ratings on how you’re doing in the various areas you’ve chosen to monitor. I prefer A through F, but you can use anything you want as long as it’s both clear and simple. Ratings will also include a composite, overall score for your progress vs. your goals.

Provide Recommendations. Finally, the system tells you exactly what to do to improve your ratings in the various areas and overall. So if you’re tracking health, for example, and you have a C in activity because you’ve been sedentary, the system will tell you what to do to improve it. It’ll give clear and prescriptive advice, such as, “Row 500 meters, do one set of push-ups, and one set of sit-ups every morning.” If you’re working on building a great team, the advice after a bad rating might be, “Have more frequent team meetings, and focus on building trust through reduced competitive focus.”

Adjustment. The last component of the system is the means by which the model can be updated. Updates to the system come in the form of modifications to the model. This can be addition, subtraction, or changes in importance for elements under consideration. For example, if you’re tracking a family’s health and happiness, a new study could come out that says shared laughter is crucial to individual happiness. This will be incorporated into the model and recommendations accordingly, based on the research. Similar adjustments will also be made to the model as new information about the world is made available to us.

The adjustment phase is where algorithms will be so crucial. Using machine learning, evolutionary algorithms, and still-undiscovered AI techniques we will continue to extract increasingly valuable insights from the data we have. And because of our access to realtime data through Universal Daemonization, the data being fed into these models will be continuous and fluid.

DOM is just a methodology—a name for a simple yet powerful concept.

Have goals
Have an approach to achieving them
Bring in data about the world
Rate how you’re doing
Recommend changes based on where you could improve
Adjust the approach based on new data
(optional and/or occasional) Ensure that your goals have not changed

This is a framework for using technology, data, and science to steward humanity’s progress forward.

Summary

It’s one thing to be able to capture data in realtime, move it around, and analyze it.
It’s quite another to be able to use that data to power your models for improving outcomes.
That’s what frameworks similar to DOM, realtime data, and machine learning and evolutionary algorithms will help us accomplish.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

The Real Internet of Things: Getting Better at Getting Better

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: Getting Better at Getting Better.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

Once we are powered by realtime data and the infrastructure that makes use of it, the intelligence of our algorithms will become paramount.

Two areas seem particularly promising: machine learning and evolutionary algorithms.

Machine Learning

Machine Learning is basically the upgrade to our previous-best method of analyzing data—statistics. Where statistics are largely static (the model for extracting truth from data doesn’t improve as you add data), with machine learning the analysis actually improves itself automatically.

Machine Learning, in other words, is the ability for computers to learn without being explicitly programmed. And when you apply that to the algorithms doing realtime data analysis of trillions of objects, we can expect the results to be truly remarkable.

We’re not just learning about the world; we’re improving our ability to learn about the world automatically. And the more data we see the better it gets at improving itself.

Evolutionary Algorithms

As excited as I am about machine learning, I’m even more excited about evolutionary algorithms—especially when they’re eventually combined.

Evolutionary algorithms work by modeling evolution’s method of improving things. It has three basic steps:

Collect lots of different things together
Combine or mate them with each other
Introduce randomness into the output
Test that output against the environment to see what wins

Another way to say that is:

Descent with Modification
Natural Selection

That means lots of varied input, combined, random mutation, and then selection of winners.

It’s important that you have a good, varied pool to start with. It’s also important that you add randomness to the output step so that completely new things are created. And finally, it’s crucial that you have a good environment to test in (one that truly represents success or failure).

In nature this is easy—it’s just the real world the organism is trying to survive and reproduce in. In the digital world it’s a bit more complex.

But the concept is the same, and so is the benefit.

The promise of evolutionary algorithms is that they will allow us to create, very quickly, solutions that human designers couldn’t possibly conceive of (and definitely not in that span of time). They work by taking simple inputs, mating them together, adding some random component, and then automatically testing the output to see how successful that generation is. The winners go on and reproduce, with some randomness, and new outputs are tested again.

This is repeated through a number of generations until the line either dies out or something successful is created.

What’s so spectacular about this is that with constantly improving hardware, combined with better ways of modeling reality, we can go through thousands or millions of generations of evolution looking for solutions to our problems, all in minutes or hours. Using this technique we can potentially outperform the creative capabilities of billions of the smartest humans, doing their best on a problem for hundreds of years, all in the span of a few hours.

Now imagine that mechanism for improvement, i.e. the one that got single-celled organisms all the way to the point of being able to explore our solar system, and combine that with machine learning algorithms trained to improve the quality of the evolutionary algorithms.

It’s difficult to overstate the benefits that can come from being able to accelerate not just our ability to learn, but our ability to learn how to learn. That’s precisely what the combination of machine learning and evolutionary algorithms can do—both on their own and when used together to enhance each other.

Summary

Traditionally the best method we’ve had for learning about the world has been statistics, which are largely static; the analysis model doesn’t improve when you get more data.
With machine learning, the system gets smarter by itself, i.e., without needing to be reprogrammed.
Evolutionary algorithms leverage the power of descent with modification and natural selection to create and test possible solutions to problems that we never could as humans.
Combining these two—with machine learning improving the modeling and testing capabilities of evolutionary algorithms—may be one of the most powerful advances in technology we’ll see for the foreseeable future.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

The Real Internet of Things: The Four Components of Information Architecture

To benefit from the work I put into my typography, read natively at: The Real Internet of Things: The Four Components of Information Architecture.

—

These are published chapters from my book The real Internet of Things, published on January 1st, 2017.

There are many different information technologies that will be invented and adopted in the coming decades, but I believe there are four (4) primary categories that they will all fall into.

Realtime Data
Data Transfer
Analysis Algorithms
Presentation Interfaces

Realtime Data

As I spoke about in the realtime data chapter, knowledge of the current state of the world is extraordinarily empowering. It allows us to ask questions about the state of the world and adjust behavior as a result. The more realtime the better, and the more standardized and usable the format the better.

Data Transfer

Now that we have the data available, we need to be able to get it to the algorithms that will perform work on it. The protocols will have to be not only standardized, but built to allow trillions of tiny queries and updates, since even one object’s various state attributes could be changing in tens, dozens, hundreds, or thousands of times per second.

Analysis Algorithms

Once we have this data the focus turns to the algorithms that will do the analysis. As we talked about in the ‘Businesses as Daemons’ chapter, companies will largely compete as data analysis algorithms. Companies will largely have access to the same data; the question will be what you can do with that same data that gives you the competitive advantage.

Presentation Interfaces

Finally we have the output step. We’ve captured the realtime data, we’ve moved it to where it’ll be analyzed in a standard and efficient way, some company has done their unique analysis on it, and now we’re going to display it to someone or something. That’s presentation, and it will be another opportunity for companies to differentiate.

Creating the ability to track and present realtime data about objects (and ultimately the world) is hard. That’s an engineering problem. The other engineering problem is creating the protocols that will allow us to constantly poll and update objects for their state changes, which will be trillions per second in any large set of objects (like a company, or a city).

Those are efficiency and scalability problems.

The algorithm and presentation steps are significantly more creativity and innovation based. They are ultimately what will differentiate competitors in a long-term business market.

There will be innovators solving the engineering problems as well, but it’s infrastructure. It’s the connective tissue that enables the competition in the spaces of algorithmic analysis and presentation of results.

There is also the option for the output of one algorithm to be sent to one or many others as well, of course.

Summary

Realtime data is collected from the world.
The data gets evaluated by algorithms.
The output of those algorithms gets presented in some useful way.
The collection and transfer of the realtime data are engineering problems, and the analysis and presentation are creative/innovative problems.

CHAPTER NAVIGATION

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

Your Mind’s Software is More Important Than Its Hardware

To benefit from the work I put into my typography, read natively at: Your Mind’s Software is More Important Than Its Hardware.

—

Many people see intelligence as a function of brain power. So if you have a bigger brain, or smart people in your family, then you’re likely to be smart yourself.

I’m sure that’s true to some degree, but I think a better model is that of a computer…where you have two components working together: 1) the hardware, which is your actual brain, and 2) the software that runs on it, which is your education and experience.

One of the most descriptive characteristics of computer software and hardware is the frequency that it gets updated. In general, hardware doesn’t get upgraded often (or ever) before it’s replaced, while software is often updated many times during a product’s lifecycle. Importantly, it’s possible to have dramatically different capabilities and performance based on the software you’re running, even if the hardware is the same.

When it comes to success in life your mind’s software is far more important than its hardware.

We each run an operating system on our brain, and that operating system comes from the combination of formal education and life experience we’ve accumulated over our lifetimes.

To me it’s critical to understand that this operating system requires constant updates to be able to best manage life in our modern and constantly evolving society. Being successful in life is about processing information and making decisions, and if the world is changing your software has to change with it. Here are some mappings from education equivalent to operating system.

Elementary School –> Windows 98
High School –> Windows XP
Bachelor’s Degree –> Windows 10
Advanced Degree –> Ubuntu
Avid Reader –> Alpha Zero / IBM Watson

These are obviously tongue in cheek, but I think they reveal metaphorical truth regarding our limitations in processing the world. Here are some numbers around opinions held by Americans, for example.

One third of Americans don’t know any branches of government, and only a quarter know all three.
Only 12% of people running Windows 98 on their brains (High school diploma or less) believe that humans evolved without God being involved in the process. The number is almost three times as high for postgraduates (Ubuntu).
A quarter of Americans don’t know what country we declared our independence from.

For most facts like these, the percentage of people holding a ridiculous belief is directly correlated to their education level. This isn’t bad hardware. It’s bad software. And to fix it you don’t need a better brain—you need a better way of seeing and processing the world.

The metaphor for operating system isn’t perfect, but the general idea is the lack of bugs, limitations, and the ability to constantly update based on new information as you get to higher and higher levels.

What interests me most, however, is not making fun of people without college educations. That’s mean, stupid, and worst of all—unhelpful. What’s interesting and useful is realizing the importance of reading in all of this. Having a college education gets you to Windows 10 level, which is pretty decent, but many of the most famous and smartest people in the world are, first and foremost, voracious readers, and that’s what gets you to the IBM Watson / Machine Learning level of understanding.

You don’t really start getting old until you stop learning. Every book teaches me something new or helps me see things differently. I was lucky to have parents who encouraged me to read. Reading fuels a sense of curiosity about the world, which I think helped drive me forward in my career and in the work that I do now with my foundation. ~ Bill Gates

When Warren Buffet was asked what he owes his success to, he pointed at a stack of books and said:

Read 500 pages every day. That’s how knowledge works. It builds up, like compound interest. All of you can do it, but I guarantee not many of you will do it. ~ Warren Buffet

This is why I use the IBM Watson / Machine Learning metaphor for people who read a lot (let’s say more than 20 good books a year). It’s because they are constantly taking in the best input available and using that content to upgrade their model of how the world works.

College doesn’t give you this. College gives you base knowledge, and hopefully some idea of how to find more information and think about new inputs. But it doesn’t supply an infinite number of new lessons and experiences.

Only reading does that.

Another way to look at that is to think about how many good ideas and good books you consume while getting a bachelors, a masters, or even a PhD in college. 20? 30? 50? The number is probably quite low if you add them up, and if you become a significant reader of good books on similar topics you’re likely to catch up and exceed that knowledge level in a very short amount of time.

Being an aggressive reader is the equivalent of having a major computer upgrade purely through software—and if you’re relying on your old-school college education you’re effectively stuck on an OS that is no longer receiving patches.

Summary

Ok, so what’s the point here? What’s the takeaway?

The human mind can be thought of as a computer, complete with both hardware and software.
The software is more important than the hardware, so running Windows 10 on a 2 CPU computer with 128GB of memory is better than running Windows 98 on 24 CPUs and a terabyte of RAM.
The smartest people in the world aren’t just the people who went to college—they’re the people who read constantly.
If you’re someone without any formal education, try to get one as soon as possible, and ensure that all your loved ones do the same. It’s the equivalent of going from a 20-year-old OS to a modern one.
To get true state of the art information processing capabilities, you must make reading a central part of your life.


Notes


There are obviously different reasons why some people are unwilling and/or unable to initiate this software upgrade of education and/or reading, but those reasons are numerous and nuanced, and are thus worthy of their own post.
Thanks to Mark Cunningham for talking through some of these concepts with me.

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

We Should Be Cautious When Building Evolutionary Algorithms

To benefit from the work I put into my typography, read natively at: We Should Be Cautious When Building Evolutionary Algorithms.

—

A while back I wrote that Evolutionary Algorithms Could be More Significant Than Machine Learning. The reason for this is that they keep improving towards a specific goal, without additional input, by leveraging the powers of evolution, i.e., reproduction, variation, heritability, and differential success.

Reinforcement Learning is considered a subset of Machine Learning by many, so this isn’t technically a one-vs-other scenario.

This means that evolutionary algorithms, properly configured, can produce extraordinary results in a very short time, and the problem is that we won’t always what they’re going to create—or what impact that creation will have on society—before it happens.

Two recent examples:

Deep Blue took over 10 years, and required many experts and millions of dollars to create before it could beat a human at Chess. Alpha Go beat the best human at Go, but it took dozens of engineers giving their experience and programming, required 140 Google CPUs, and took hundreds of hours to finish. Alpha Go Zero beat Alpha Go in 3 days using only 4 CPUs and didn’t require any initial training whatsoever.
Facebook uses many of the concepts of an evolutionary algorithm to constantly adjust content and UI/UX to make people spend more time in the platform. We thought we were building a useful social media platform, but what we actually built was a giant sinkhole of human attention that seems to be causing significant mental health issues.

Bret Weinstein talked about this point of social media and addiction situation on Sam Harris’ podcast.

The obvious and much discussed case is Artificial General Intelligence (AGI), which will be able to use these same techniques to improve itself very quickly, but there are many more present, near-term, and realistic examples that we need to watch for as well.

The takeaway here is quite simple: we need to be very careful about creating anything that both self-improves and has significant interaction with humanity. Failing to do so can product all manner of harm to us, from an annoyance (really good advertising) to catastrophe (AGI without morality).

Use. Evolutionary Algorithms. Wisely.

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

Analysis of Sam Harris’ Podcast With Bret Weinstein

To benefit from the work I put into my typography, read natively at: Analysis of Sam Harris’ Podcast With Bret Weinstein.

—

One of the few podcasts that I follow…religiously…is Waking Up, by Sam Harris. As a fan of Sam’s from the beginning of his public life, and someone who’s had a number of email exchanges with him over the years, I think his podcast format is just spectacular. He does one guest per show and takes them through where they agree, where they might disagree, he gets them to talk about their main topics they care about, and then gets them to comment on various important issues happening at the time.

Episode #109, with Bret Weinstein, was one of my favorites in a long time—both because of the quality of Bret as a guest (he’s an Evolutionary Biologist) and because of the level of interest I have in the topics they were discussing. The episode was so good that I decided to listen to it again, while taking notes, and this post will be me capturing various parts of it and commenting on them.

Some of this will be direct quotes, other parts will be me paraphrasing, and some of it could be somewhat incorrect since I’m not an expert in Bret’s field.

You don’t have Inuit marathon runners because they’ve been selected to retain heat, whereas Ethiopians and Kenyans have been selected to dissipate heat, and this should not be surprising or controversial.
Culture is units of information transmitted from one member of a species to another.
Humans dominate because we’ve offloaded more of the day to day processing required for success from our computer’s hardware (the brain) to the software (culture).
Humans are evolutionarily odd.
We are the most nurture-based creatures in history.
Our success is determined by nurture far more than any other creature on earth, and this is because it provides a distinct evolutionary advantage.
Humans are basically so amazing because we do so much more in software (which is easier to be upgraded) than in hardware.
Culture can move horizontally, but it normally moves vertically as it’s passed down from parents to offspring.
The cultural piece is every bit as biological and evolutionary as the genetic layer, and it’s a special trick deployed by the genetic layer to solve problems not solved by genetics alone.
Intersectionality is where each group who has some claim of oppression or discrimination against them has a special flavor to it, and the more of those groups you fall into (black, trans, etc.) the higher rank you are in a new, hypothetical world where oppressed people are in charge over their previous oppressors. The basic idea is that each group has some sort of discrimination against it, and if you’re in more of those groups then they stack up. Sam compared this to being a D&D party where you botch your spellcasting and harm the party.
Birth control changes everything in gender roles and gender politics.
Men and women are different. The biggest thing (from Bret) was that women are invested in their children because they can’t keep making kids. Males can keep having kids, so they’re tuned to keep taking risks to have more kids.

We have to grapple with the fact that the academe is producing people who not only know nothing, but who believe things that make them unable to learn.

And they are teaching others. https://t.co/ztCEGwoFAQ

— Bret Weinstein (@BretWeinstein) December 21, 2017

It’s not true that every non 50/50% profession has some measure of discrimination.
Gender is the software of sex, and sex is the hardware.
As a biologist, gender is interesting, but as a human the answer is simple: we have to have compassion.
Metaphorical truth is a belief that is factually wrong but you come out ahead if you believe in it.
Religion was valid metaphorical truth because it helped us before (for thousands of years). We’re now at a point where it (often) hurts more than it helps, but that doesn’t change how useful it was in the past.
Sectarian differences in doctrine is basically the same as evolution with mutations, and it’s not that they’re all fighting to win, just as it’s not true that all species are fighting to win. It’s more like they will all be tuned for their times and their environments. So we will co-exist being the best adapted species for that time and place.
The reason genetics has moved to using culture and memes is because it moves so incredibly fast.
Bret thinks kids are just as smart at birth, but that when they’re adults there are cognitive differences. I think that’s a nice idea.
Bret thinks that we need to be extremely cautious because of four characteristics that create adaptive evolution (reproduction, variation, heritability, differential success) which is such a powerful force that we cannot keep unleashing it on ourselves. An example is an evolutionary upgrade of social media sites that optimize for addictiveness. We created this system that adapts to get better and better at holding our attention, and now we have a societal problem with social media addiction.
His point is that you can’t simply make these systems and release them into the world because they’re so incredibly powerful and we’re not smart and thoughtful enough to know what they’re going to optimize for. And because we didn’t know what they’re going to do we obviously couldn’t have a smart conversation about whether or not this is good for us.

What you are describing is like one unnecessary regulation away from cake-utopia. https://t.co/Cr0TeUD20n

— Bret Weinstein (@BretWeinstein) December 25, 2017

If you stop eating farm animals they go extinct.
Wisdom requires delayed gratification. You have to go carefully.
The same parameter that makes a mother love her children also makes them commit genocide.
We have to understand that we’re all running according to this code and find a way to navigate.
We’re going to need drugs to figure out how to get out of our current predicaments because the current contents of our consciousness don’t seem to be enough.
We have to give up on evolution’s purpose for us, because it’s identical for everyone and bad. If you have the same purpose as a liver fluke, and a malaria virus, and a fig tree, we have to recognize that we need to evaluate that purpose and reject us. Good people want to live in a safe, anti-fragile society where we’re free to live their lives in a meaningful way.
Bret is encouraged that the Libertarian Left (liberty, not economics) will start coming out to oppose the Authoritarian Left that’s basically hijacked the non-right part of the spectrum in recent years.

There are a lot of great ideas here, but my favorites are:

Human success is so magnified because we’ve offloaded work from our brain hardware to our culture software.
That culture is passed down from parents to children, and helps a given group succeed even more than their genes.
The biggest difference between men and women is that women biologically have to invest in the children they have (because they can’t make more after menopause) whereas men can just keep taking risk to keep having more and more kids. This probably explains a lot in our society.
Biology is the hardware of sex, and gender is the software.
Metaphorical truth is untrue but useful, but (like religion) it eventually becomes outdated and starts doing more harm than good.
We need to watch out for any system where we release adaptive evolution because it will optimize for SOMETHING very quickly, and we are usually in such a rush that we either don’t know what that something is and/or we won’t have time to discuss whether it’s a good idea.
Evolution’s purpose for us is bad. It’s basically to survive and reproduce at the expense of everyone else. And it’s the same for viruses and trees and maggots. This is not any way to live a life, so we have to find something to replace it with.

Such an unbelievably rich podcast in terms of ideas, good faith, and potential for a better world.

I highly recommend subscribing to Sam’s podcast. Episodes like this make it absolutely worth it. Now I’m off to listen to his brother’s podcast that aired a bit before this one.

Notes


I went to the live one with Eric and Sam in SF as well. Quite good.

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel

Two Kinds of Human Meaning and Enjoyment

To benefit from the work I put into my typography, read natively at: Two Kinds of Human Meaning and Enjoyment.

—

The idea is that there are two ways for humans to derive happiness:

This will be a stream of consciousness thought process that will likely lead to an essay later.

By following tradition and ritual
By being curious and evidence-based

Humans have long been sustained by number 1. It’s what family and religion are based around, and to follow such doctrines and drives tends to produce the most powerful types of pleasure and fulfillment in us.

It’s only been the last couple of thousand years that we’ve started to derive great pleasure from questioning things, learning things, and searching for truth using reason rather than dogma.

This break from tradition has brought us great gifts, such as large societies, science, a greatly improved lifespan, less war, etc., but it’s also injected something non-ideal into the human psyche.

Doubt.

When you know God loves you, that your kids just need to be Christians or Jews or Muslims who marry others of the same kind, go to the same churches, act the same, eat the same foods, etc.—and all will be well, that’s comforting.

Following that template while having kids and growing your family gives a feeling of safety, and security, and bedrock.

It also produces the worst types of hatred and war because different groups of people inevitably have different sets of dogma, and since they’re all convinced theirs is correct they’re destined to go to war about it.

And then there are the groups who have discarded supernatural belief and dogma. They are atheists and agnostics who realized that the old ways were factually incorrect and the cause of extraordinary hatred, suffering, and bloodshed.

Their problem is that it’s quite hard to replace religion and dogma and tradition.

Yes, you can receive great joy from discovery, and from knowing you live a truth and equality based life. But it’s also more empty. It’s less fulfilling in some fundamental way.

Humans love ritual. They seem to need it. Tradition, ritual, patterns, absolute beliefs—these all produce profound happiness in us, and discarding them can be detrimental to both individuals and societies.

For me the obvious answer is that we have to move away from the first and towards the second, but because of what I’ve written above I think it’s crucial that we understand that we lose something when we do. We must find a way to maintain some part of the ritual/tradition system in the new curiosity and exploration one.

We have to find a way to give structure, and confidence, and ritual to the fundamental values of truth and equality and reason.

To ignore this requirement is to repeat the same mistake over and over again—pulling people from the pit of old-style religion only to offer little in return. This produces dissatisfaction and resentment in some, and a feeling of resigned emptiness in others.

We can and must do better.

We have to merge the systems into something superior to both.

—

I spend 5-20 hours a week collecting and curating content for the site. If you're the generous type and can afford fancy coffee whenever you want, please consider becoming a member at just $10/month.

Begin Membership

Stay curious,

Daniel