Daniel Miessler's Blog, page 75

The Wonder of Entropy

I find entropy fascinating—especially the relationship between it and information.

I just read this book about Claude Shannon, who invented information theory, and it basically unified everything I’d known about it previously.

Here’s how I’d capture my current understanding:

Entropy is basically a measure of disorder
For the universe, the ultimate disorder is heat death, meaning there are no patterns at all in the atoms of the universe. It’s cold chaos.
Information is the opposite of disorder and chaos. It has patterns that convey something to the receiver.
The more organized something is, the more predictable it is—because when you make a move it limits the moves you can make next according to the rules you’re using For example, writing “xyl” severely limits what you can write next to construct a valid English word.

This is why entropy is used in computer security.

When we say something has high entropy, it means it is highly chaotic. That it doesn’t contain discernable patterns. And ultimately—that it’s relatively unpredictable.

Like Heat Death.

If you were floating around in a post-heat-death universe, and you looked at a stream of atoms in any direction, you would see a completely unpredictable sequence. And that’s quite opposite from looking at a stream of atoms within a star, or a planet, or even in a dust particle. Those atoms are organized.

The other related facet of this is the concept of surprise. Information should be high in surprise, because it’s telling you something you don’t know. And you can’t know what’s going to be said based on what was said before.

So if someone says, “I regret to inform you, but we were unable to…”, we have low entropy in the series “I regret to inform” because we are almost guaranteed to receive a “you” after that. But we don’t know what comes after the “unable to”, so that’s information.

I’m not an expert on this, but it seems each scale and segment of a message can have its own entropy, e.g., within a word, within a sentence, within a paragraph, or within a book. Maximum entropy is maximum surprise, since the next bit is completely unpredictable.

I find this relationship between entropy, patterns, and surprise both remarkable and fascinating.

Anyway.

Whether it’s the decomposition of a living creature, the return of a solar system to the state of stardust, or the creation of a pseudo-random string to be used in a security context, the core principle of entropy is disorder and unpredictability.

Summary


Entropy is a measure of, or motion towards, disorder.
The more chaotic something is the less predictable it is.
Surprise is the opposite of predictability.
Information can be measured in surprise, i.e., the amount of data conveyed that wasn’t expected.
Entropy in computer security is about producing sequences that have little to no discernible pattern, i.e., that are unpredictable.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

The Difference Between System V and SystemD

One of the most fundamental distinctions in modern Linux systems is whether they use SystemV or Systemd. Here are the main differences between the two.

SystemV is older, and goes all the way back to original Unix.
SystemD is the new system that many distros are moving to.
SystemD was designed to provide faster booting, better dependency management, and much more.
SystemD handles startup processes through .service files.
SystemV handles startup processes through shell scripts in /etc/init*.


Indicators


If you’re starting and stopping things using systemctl restart sshd, etc, you’re on a SystemD system.
If you’re starting and stopping things using /etc/init.d/sshd start, etc, you’re on a SystemV system.


Which distros use which?

Many older versions of SystemD distros were SystemV.

Here’s an incomplete but hopefully useful breakdown of which distros are on which system.

Systemd: Amazon Linux, Red Hat Enterprise, CentOS, Fedora, Debian/Ubuntu/Mint

SystemV: Gentoo, Alpine, Slackware, Linux from Scratch

Notes


There are other startup systems as well, such as Upstart and BSD.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

My Collection of Statistics About Americans

Every month or so I am astounded by some statistic related to the American population. I always tell myself I’ll capture them somewhere, and this is that place.

55% of Americans are not invested in the stock market in any way. More
58% of Americans have less than $1,000 in savings. More
40% of Americans would struggle to come up with $400 for an unexpected bill. More
American debit in 2018/2019 averaged $136,365, and totaled $13.95 trillion. More
Americans have an average of $6,849. More
The Median (half above, half below) household income for Americans in 2018 was $63,179. More
12,808,000 American lived in poverty in 2017, which was 17.5%. More
21 million children receive free or reduced cost lunches at school. More
45% of Americans believe in the existence of ghosts and demons. More

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Unsupervised Learning: No. 205

podcast and newsletter every week instead of just twice a month.

Exploring Both Sides of the Free Speech and Cancelling Debate

With all the both-sided vitriol around free speech, canceling, and de-plaforming I thought it a good time to inject some equally both-sided common sense.

Basically, one side thinks it’s wrong to de-platform people for having controversial—but not directly violence-inspiring—viewpoints. They often quote the first amendment about how nobody is supposed to ever be allowed to silence anyone on their platforms.

Except, that’s not what the first amendment says. It says that “congress” shall make no law, meaning, the government. It has nothing to do with regular folks, which is what we’ll talk about next.

Ok, so the restrictions that exist are really about government silencing people, and people saying things that can directly cause violence or harm to others.

So does that mean companies should be allowed to kick anyone they want off their platforms?

That’s what the other group says, which is basically that someone spewing unpleasant or stupid or hateful ideas can be kicked out of a local flower shop by the owner—so there’s no difference between that and a social media company kicking someone off their platform for vaccines are dangerous.

That makes sense.

The problem is when these two things combine.

Let’s do that by imagining a giant social media company called—well, let’s just make up a name. Let’s call it “Face book”.

So let’s say Facebook beats out all its competitors and becomes the one and only way for people to be heard at a large scale. And let’s say it’s run by some really, really rich Republicans who hate gender equality, climate change politics, and are spending billions telling people to avoid vaccines.

They also hire thousands of people to write algorithms and manually filter content on their platform, and they make sure nobody is saying anything positive about these things. When they see them, they just delete them.

So a whole bunch of liberals start getting upset. They say they’re being shut out. They say they have no voice.

And the Republicans respond by pointing to the street corner, and saying, “Nobody’s stopping you from your free speech. Feel free to assemble on the street, start a blog, or put a sign on your front lawn. Nobody’s stopping you.”

Now here’s the question: is that ok? Should society simply shrug and walk away from this?

What are the grounds or mechanism for stopping them from excluding content? They’re just a flower shop, right?

No, that’s not right. They’re not a flower shop. They’re a speech shop. And they’re the only speech shop.

The moment things changed was the moment they became the only voice that could reach the masses. Sure, you could technically send postcards or try your hand at HAM radio, but effectively—in reality—you’re dead if you’re blocked from the one and only social media network in the world.

My argument is that we need to look at the spirit of the constitution, not only the letter.

It says “congress” because it’s differentiating between the overwhelming force vs. the little guy. People are the little guy. Getting on the street corner is the little guy. Standing on soap boxes is the little guy. And so are flower shops. Those folks are free to ban you all they want, for whatever reason—because they’re not in charge.

That’s the difference. Having control.

Once you have control—according to the spirit of the constitution—at least as I interpret it—you take on the responsibility of impartiality.

Right now we only have a couple of giant media companies—Facebook, Twitter, and YouTube, and since they’re mostly banning the same people I’d argue we’re getting pretty close to the situation I argue the founders were trying to avoid.

And that raises an interesting question.

Does that mean that the government will then have to become the counterbalance to the biased monopolies? So in this case, the government would have to have a social media PBS show that features the liberal side of the issue?

Or does the government simply step in and create a law saying FaceTube has to do that?

Either way, I hope this has helped you think about the situation differently.

It’s bad if the government can silence opinions, which is why we made a law against it. But if you look at it closely, the law is really about overwhelming forces silencing opinions—and they only mentioned the government because they hadn’t yet heard about FaceTube.

Thanks to Jeremiah Grossman for talking through this with me, and for coming up with the idea that the government might have to (ironically) come up with a PBS of ideas if the corporate world can’t handle the ask.
Just in case any extreme right person thinks I’m secretly saying FaceTube should allow their drivel, you’ve got the wrong tree. The most American thing the US has ever done was defeating the Nazis. And I voted for Obama both times. This isn’t about one side vs. the other; it’s about how to find balance and conversation given the fact that technology is changing all the rules.
Just in case any extreme left person thinks I’m saying all extreme right stuff should be allowed to stay on FaceTube, or should absolutely being kicked off, I’m actually not saying either. I’m saying that the First Amendment was a great law that needs to be re-interpreted given technology and the danger of propaganda today. We have new forces at play that change the calculus.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Exploring Both Sides of the Free Speech and Cancelled Debate

With all the both-sided vitriol around free speech, cancelling, and deplaforming I thought it a good time to inject some equally both-sided common sense.

Basically, one side thinks it’s wrong to deplatform people for having controversial—but not directly violence-inspiring—viewpoints. They often quote the first amendment about how nobody is supposed to ever be allowed to silence anyone on their platforms.

Except, that’s not what the first amendment says. It says that “congress” shall make no law, meaning, the government. It has nothing to do with regular folks, which is what we’ll talk about next.

Ok, so the restrictions that exist are really about government silencing people, and people saying things that can directly cause violence or harm to others.

So does that mean companies should be allowed to kick anyone they want off their platforms?

That’s what the other group says, which is basically that someone spewing unpleasant or stupid or hateful ideas can be kicked out of a local flower shop by the owner—so there’s no difference between that and a social media company kicking someone off their platform for vaccines are dangerous.

That makes sense.

The problem is when these two things combine.

Let’s do that by imagining a giant social media company called—well, let’s just make up a name. Let’s call it “Face book”.

So let’s say Facebook beats out all its competitors and becomes the one and only way for people to be heard at a large scale. And let’s say it’s run by some really, really rich Republicans who hate gender equality, climate change politics, and are spending billions telling people to avoid vaccines.

They also hire thousands of people to write algorithms and manually filter content on their platform, and they make sure nobody is saying anything positive about these things. When they see them, they just delete them.

So a whole bunch of liberals start getting upset. They say they’re being shut out. They say they have no voice.

And the Republicans respond by pointing to the street corner, and saying, “Nobody’s stopping you from your free speech. Feel free to assemble on the street, start a blog, or put a sign on your front lawn. Nobody’s stopping you.

Now here’s the question: is that ok? Should society simply shrug and walk away from this?

What’s the grounds or mechanism for stopping them from excluding content? They’re just a flower shop, right?

No, that’s not right. They’re not a flower shop. They’re a speech shop. And they’re the only speech shop.

The moment things changed was the moment they became the only voice that could reach the masses. Sure, you could technically send postcards or try your hand at HAM radio, but effectively—in reality—you’re dead if you’re blocked from the one and only social media network in the world.

My argument is that we need to look at the spirit of the constitution, not only the letter.

It says “congress” because it’s differentiating between the overwhelming force vs. the little guy. People are the little guy. Getting on the street corner is the little guy. Standing on soap boxes is the little guy. And so are flower shops. Those folks are free to ban you all they want, for whatever reason—because they’re not in charge.

That’s the difference. Having control.

Once you have control—according to the spirit of the constitution—at least as I interpret it—you take on the responsibility of impartiality.

Right now we only have a couple of giant media companies—Facebook, Twitter, and YouTube, and since they’re mostly banning the same people I’d argue we’re getting pretty close to the situation I argue the founders were trying to avoid.

And that raises an interesting question.

Does that mean that the government will then have to become the counterbalance to the biased monopolies? So in this case, the government would have to have a social media PBS show that features the liberal side of the issue?

Or does the government simply step in and create a law saying FaceTube has to do that?

Either way, I hope this has helped you think about the situation differently.

It’s bad if the government can silence opinions, which is why we made a law against it. But if you look at it closely, the law is really about overwhelming forces silencing opinions—and they just mentioned the government because they haven’t yet heard about FaceTube.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

The Frustrating Potential of Eric Weinstein and The Portal

I’ve been following Eric Weinstein ever since seeing him live with Sam Harris in San Francisco a few years back. He has the trifecta of gifts in the form of kindness, intelligence, and honesty, and that’s seriously refreshing to have at the moment.

Unfortunately he’s driving me insane right now.

He just had Sam on his show, The Portal, and I seriously turned the epsisode off and back on again three times.

Here’s my attempt to capture what I found so unpleasant.

Eric didn’t allow Sam’s points to land. He (mostly) either said Sam was wrong outright, or came over the top with something like, “I’m surprised we’re disagreeing this much.”, or “I can’t believe you’re seeing this different than me.”
He seems too infatuated with the beauty of his own ideas and his own sentences. It doesn’t mean he is, but that’s how he comes off.
He’s constantly making points that are almost right—but not quite.
He does the exact opposite as well, which is take a perfectly made point made by Sam and say, “Maybe”, or “Almost”, or some other response that refuses to give due credit.

So the big thing on this episode was the idea that Trump was empowered by a wholesale selling out of the lower class by the elites. He seems to believe this to a nearly conspiratorial level.

Basically, he thinks that the leftist elites put a ton of policies in place that ultimately destroyed the middle class and are now crushing the working and poor classes. And he thinks the people inherently feel this, and that they’re punishing the left for it by electing Trump.

Sam and I see the world as far more complex than that. Is it true that some liberal policies actually ended up hurting the middle class and poor people? 100%. But if you take a policy like free trade, or immigration, I don’t see how you can imply that their actual purpose was to hurt the working class.

Does immigration put pressure on native workers? Of course it does. Does it have benefits? Of course it does. Are there people on the left that secretly want to hurt native workers with immigration so that they can make more money? Sure. I suppose so. But aren’t there far more liberals who either don’t know that’s happening or wish that it weren’t?

It just seemed like Eric was flirting really closely with a fully conpspiratorial line around major liberal policies being secret attacks on “the people”, and I just cannot get behind that.

And he wouldn’t listen to Sam when he’d ask him to see the other side. Sam was agreeing that there could be truth in what he said, but that that fact could be diminished by more intention on the other side, i.e., there being far more people thinking they were doing the right thing as opposed to participating in a conspiracy.

But Eric wouldn’t let a single point land.

He either sidestepped and let the point glance off, or he’d catch it and come back with some version of, “I can’t believe you’re that naiive”.

The disorienting part of this is that I believe Eric is being 100% honest and acting in good faith. That’s the only reason I was able to turn the podcast back on again after quitting it two or three times out of frustration.

Listening to him is so often a tortuous combination of being slightly wrong, imprecise, and genius all at the same time. I always want more and want it to end at the same time.

It’s like spending two hours trying to take that deep inhalation to catch your breath, but never getting it—even though you’re not even tired.

I felt even more responsible to write this because he was genuiniely asking Sam for help on his show. He wa asking what he could do better.

Well, I have some answers, Eric, and I’ll be DM’ing them to you as well since we follow each other on Twitter.

Let other people’s points land fully; I know you’re acting in good faith, but others won’t when you just walk through their reasoning
Try to be super concise when making a clarifying point; don’t allow yourself to build a fragment into stream of consciousness
Focus on the impact of the idea and not on its structure or sound. Your language is beautiful, and your intelligence is beyond question, but the game is actually convincing the other person through empathy
As a rule, if it sounds overly smart it’s probably too opaque: notice how Sam’s sentences are usually quite simple. They add up over time through layered simplicity.
Remember that a big part of what people need from the portal is clarity; it doesn’t have to be simple, but it does have to be transparent

My bottom line here is that The Portal is an awesome idea, and Eric is an awesome intellectual who I am thankful to share time and a planet with.

We just need more clarity and transparency in your points, my friend.

Tight
Crisp
See their side
Acknowledge it
Refine your point to address the disconnect

We’re here supporting you.

Notes


One thing I worry about is if some or all of the things I mentioned here are a result of Eric’s disability, and thus something he’s struggling with and trying to fix. If that’s the case, well, then I’m an asshole and I apologize. Eric Weinstien at a 4/10 is anyone else at a 9/10, so don’t stress it.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Unsupervised Learning: No. 203

podcast and newsletter every week instead of just twice a month.

A Beginner’s Notes on Meditation

I’ve wanted to get into meditation since I was in my late teens, but it just never materialized. Everything I did learn seemed too ephemeral and distant, and I lacked the time to go deep enough to get past that.

I’ve tried multiple meditation apps and Sam’s is my clear favorite.

This has finally changed with the benefits of technology—namely podcasts and meditation apps. I am particularly fond of Sam Harris’ Waking Up app (and book of the same name), and it was his consistent exposure to the practice that finally pushed me through.

I am not an expert in mediation, and in fact I’m not even a decent novice. But I already feel as if I’ve learned enough to capture, so I’ll dedicate this page to what I learn—keeping in mind that I may get it all wrong early on (or perpetually).

Stoicism also teaches us to separate the input from our reaction to it.

I happen to also be studying Stoicism right now, and I find the similarities between its teachings and of Vipassana meditation to be significant and extraordinary. So I’ll be talking about that as well.

The format will be me teaching myself, all the while knowing that I may need to change it as I learn more. I hope my floundering is useful to someone.

1. You are not your inputs, and everything you experience is an input

I think this might end up being the most important and powerful concept in all of meditation. It’s the idea that every single thing streaming into your consciousness is separate from you.

Sights, sounds, pain, pleasure, thoughts, ideas, emotions, desires, etc—they’re all inputs into a very specific place called consciousness.

It seems the entire enterprise is based on this realization.

2. At any given time, you are either aware of inputs, or you are merged with your inputs

Modern CBT actually has its roots in Stoicism.

Building on #1, the second thing you have to realize is that a lack of awareness of this distinction can cause you to become those various inputs. This is very similar to what the Stoics talk about (and modern-day therapists as well) when they say there’s a huge difference between what happens to you, and how you choose to react to it.

If we fail to notice an input, and label it as such, it strikes us from the side and takes control. Only when we face it head-on can we remove its power over us, which we do through awareness.

Once you realize that thoughts, emotions, and sensory inputs are all equal in that they’re external invaders of consciousness—you realize that the problem is actually distraction.

Distraction is the default state that makes us believe we are the same as our inputs. It’s how we all live our lives. And distraction is what allows us to be hijacked by inputs.

In astronomy if you want to see something dim you have to look away from its center.

If your ankle really hurts, and you don’t view that pain as an input to consciousness, you will become that pain.
If you just gave a really bad presentation, and you don’t view your regret as an incoming and external input to consciousness, you will become that regret.
And if you are lost in thought—not realizing that random thoughts are just additional inputs into consciousness—then you will become those thoughts.

In short, the state of being distracted by life is what allows you to be hijacked by various thoughts, feelings, and emotions.

And mindfulness is simply flipping the switch from distraction to attention, which lets you see all of life’s stimuli as equal inputs into consciousness.

Observations, Notes, and Questions

Here I’ll be capturing my questions and observations as I learn more about meditation and related topics. If I make any major discoveries—or observations that I think to be a discovery—I’ll update the section above. Everything else will just be captured questions and note-taking.

Nov 16, 2019 — What about focus? Isn’t that a good thing to go into a flow state, or “the zone” when working on something? Programming, for example, is extremely pleasurable when you get into a phase of pure creation. It’s almost like Transcendental Meditation—which is not as good as Vipassana—but is still better than being distracted by multiple things at once. Is this also something to abandon? I don’t think so, but I’m open to ideas.

Nov 16, 2019 — I’m confused about what the self actually is if we accept that the sphere of consciousness is not us. Is that correct? Sam talks about “the illusion of self” so perhaps he’s saying there’s literally no self at all. I wouldn’t mind that for the same reason that I don’t mind that free will is an illusion, but I do need to reconcile the difference between the perceived self, i.e., the practical self, and the self that disappears once you reduce reality to the Sphere of Consciousness. My problem is that I’m always trying to upgrade myself. I’m studying for a reason. I’m reading for a reason. I am trying to improve my own condition so that I can improve the condition of others. Those all require that there is a thing that’s being upgraded—at least conceptually. So I need to explore that and lock it in conceptually in my mind.

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.

Unsupervised Learning: No. 202 (Member Edition)

This is a Member-only episode. Members get the newsletter every week, and have access to the Member Portal with all existing Member content.

Non-members get every other episode.

Sign in

or…

—

If you like my content, you can support it directly for less than a latte a month ($50/year) which also gets you the Unsupervised Learning podcast and newsletter every week instead of just twice a month.