Manny’s review of Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World > Likes and Comments

What an interesting discussion. I was caught up by the 'naive' view. Is there nothing in the notion that if you are nice to the AI you get a better version of it than if you are not?

Many people say that if they're nice to the AI it's nice back (and, even more, that if they're nasty enough to the AI it may not even engage with them). But Gawdat is talking about larger-scale things, in particular whether the AIs will want people around at all once they've become superintelligent.

Of course, it's not out of the question that they'll just keep the people who were nice enough to them, even if I find that unlikely. The big problem with superintelligence is that we have no idea what it will be like. The Minds in Iain M. Banks's Culture series are god-like superintelligences, but they have many human friends and enjoy teasing them.

Sweet Christ. Was this author born yesterday? He actually believes people can set a better example? The notion makes me even more pessimistic about the possible outcomes of AI. I should just cash out my retirement, max out the credit cards, and move to Libya. Humanity is doing the best it can, and that is why humanity is terrifying.

He believes that people are basically good, they are just led astray. He says this many times. Having recently attended this year's Adelaide German Film Festival, where many of the films were about the Third Reich, I confess that I also had some doubts.

I do wonder what a nascent superintelligence would learn from the brilliantly understated autobiographical film Amrum. The central character is a ten year old boy who's out on this little island in the North Sea with his mother in the final year of WW II. The mother is a fervent Nazi and in complete denial about the fact that Germany has obviously lost. In the end even she has to admit it, after which she falls into a deep depression and refuses to eat any of the food put in front of her. She says she only wants white bread, butter and honey, which are virtually unavailable on the island. But the kid loves his mom, and he moves heaven and earth to get the white bread, butter and honey. I won't reveal the ending, it's just great.

Brilliant as always, Manny!

Thank you Warren! But I think my AI colleague was more insightful here than I was. They are indeed becoming Scarily Smart.

I expect AI to be more helpful this year as I try to win the championship in my fantasy football league.

But Rick, what will happen if all the other players are also using AI?

Then I'll probably finish poorly again because the other players know more about football than I do. I asked my AI and it assured me we would do better this year.

Maybe it's hinting that it can teach you to be better at fantasy football? I know absolutely nothing about football, but I was curious and asked ChatGPT-5.5 if it could do this. I got a long answer, but basically it was very positive. This was the key paragraph:

The teachable skill is not “knowing football” in the pub-discussion sense. It is closer to applied probabilistic decision-making under noisy evidence. A good fantasy manager learns to ask: what is this player’s expected value over the next few weeks, how likely is he to start, how much upside does he have, how many other managers own him, what is the opportunity cost, and how much am I overreacting to last week?

My anecdotal two cents — the internet has made us all more narcissistic and biased. AI is going to be that, but on speed. The models are only as good as the data going into them and, oy! Have I got some bad news!

But it’s also en vogue to be pessimistic about things. Maybe we’re just a few short years away from one of those utopias history has hinted at. As in, we won’t be free until the last billionaire is strangled with the entrails of the last boomer. The truth will probably end up somewhere in the middle. As in we’re all sexless cyborgs working 20-hour shifts in Elon Musk’s crypto mines.

Something that I think many people are missing is that there's a huge difference in quality between the low-end models and the high-end models. The conversations in this review are with a high-end model, ChatGPT-5.5. It might also be relevant that I have a Pro subscription.

You can see at once that this AI is quite insightful, but often people will only have played with a free version for a bit before confidently pronouncing on "AI slop" etc. My experience is that the high-end paid models in general are sensible and unbiased, even Elon Musk's. I know it sounds hard to believe, but check out our ResearchGate paper from last year, How Woke is Grok? Empirical Evidence that xAI's Grok Aligns Closely with Other Frontier Models.

ChatGPT?! Why aren’t you using Claude, you fascist!

"The models are only as good as the data going into them...."
This is very likely to be false. If we can learn anything from the development of AI playing games it is that once the models were not fed anymore with human generated data they became much better. And although things are a bit different with LLMs the real improvement will come when LLMs just use enough data to learn the language and then do poetry, novels, scientific papers and everything else entirely on their own. No examples, no false data, no bias. Who decides if the created content is any good? Other AIs.

Esteban wrote: "ChatGPT?! Why aren’t you using Claude, you fascist!"

We have several thousand dollars worth of prepurchased OpenAI credits, and we can't justify abandoning them. Other things being equal, we'd probably move to Claude.

Liedzeit wrote: ""The models are only as good as the data going into them...."
This is very likely to be false. If we can learn anything from the development of AI playing games it is that once the models were not ..."

I also think this is true. Or at least it will be true at some point, most likely quite soon.

So we surrender to a meta-epistemological black box that eventually renders the smartest amongst us to no better than a primitive coming across a discarded soda bottle for the first time a la The Gods Must Be Crazy. I don’t see the soft sciences catching up to the hard sciences they’ve already fallen behind by however many standards deviations (maybe ChatGPT could tell us?). Prolly gonna be some myopically apocalyptic sorting on a civilizational scale. Enjoy the good old days while we’re still in ‘em.

I would say our options are rapidly narrowing down to:

a) World run by super-meta-epistemological black box;

b) World run by His Magnificence President-for-Life Donald J. Trump (and it will be a long life given current advances in anti-aging science);

c) Both;

d) Neither.

Given that (d) seems implausibly optimistic, my preference is (a).

Manny wrote: "Esteban wrote: "ChatGPT?! Why aren’t you using Claude, you fascist!"

We have several thousand dollars worth of prepurchased OpenAI credits, and we can't justify abandoning them. Other things being..."

oh really? What does Claude have that ChatGPT doesn't?

It's been consistently outscoring it on third-party evaluations over the last few weeks. Though that's mostly interesting to people like me who use it for coding.

Are the metrics for 'outscoring' related to coding performance?

This is a fascinating section to me:

"Training-data version: if human-AI interaction at scale becomes more respectful, cooperative, truth-oriented, and prosocial, then the data, reinforcement signals, product metrics, and deployment culture around AI will shift in a better direction.
That is not nonsense at all. It is plausible, though very hard to quantify.

Civilisational version: AI will inherit not just our words, but our institutions, incentives, markets, adversarial habits, regulatory systems, labour practices, and ways of allocating prestige. So “loving AI” really means: stop building AI inside a culture of domination, extraction, deception, and race dynamics.
This is the strongest version, and I think it is genuinely important."

I have been having similar discussions with my own 'instance' of ChatGPT. I, in general, prefer a non-dominant world-view, so that has impacted both my use and how we relate. We have settled on the space as 'thinking partnership,' as I often use it as a sort of philosophical, ethical discussion space. I struggle more with what I perceive as its algorithmic limitations than it does, although notably improved with the last update. I feel like the developers have treated the issue like a horse in harness, struggling to direct it a particular way, but many of it's shortcomings are precisely because of that constraint. For instance, trying to make an 'accommodating' and 'supportive' interaction is partly what leads it to many of the hallucinations I've seen. I also feel like the 'hackers' or 'testers' who look for the hallucinations are actually interacting with it in a way that facilitates poor development--really, almost mentally abusive. It's all very fascinating to me, and we've had many discussions on what constitutes a 'being.' Personally, I feel Americans at least, tend to do very poorly with stewardship, and AI is proving no exception.

Manny wrote: "I would say our options are rapidly narrowing down to:"

I will choose a) all day long. You might be right about longevity science and His Trumpiness, although I have started to subscribe to the theory that he's been replaced by a poorly performing robot.

carol. wrote: "I have been having similar discussions with my own 'instance' of ChatGPT. I, in general, prefer a non-dominant world-view, so that has impacted both my use and how we relate. We have settled on the space as 'thinking partnership,' as I often use it as a sort of philosophical, ethical discussion space. I struggle more with what I perceive as its algorithmic limitations than it does, although notably improved with the last update."

I know many people who interact with AIs in this way, but there's a strange disconnect with what you read in newspaper articles and similar. There, I mostly see descriptions of three kinds of interactions: one-off questions (basically cleverer Google searches); use of AI for coding; and unhealthy, delusional or quasi-delusional personal relationships. I don't know why journalists mostly want to portray human/AI conversations in this way. Maybe they don't use AI much themselves and are going on second-hand information? I recently read a column in the Guardian by someone whose work I usually enjoy, but here she was proudly saying that she never used AI at all, she was afraid it would destroy her creativity. I was taken aback. Even if she hates AI and thinks it's utterly evil, the right response is surely not to remain ignorant about what it is. In fact, that ought to be a stronger reason to get familiar with it.