A memorable case: on what was to be one of the last normal weekends in 2020, I took an Uber with a friend to an event. On the way we discussed the rising warnings of an international pandemic and our preparations. But my friend wanted us to talk more discreetly, lest we scare the Uber driver.

Why on Earth would it be bad to scare the Uber driver? My friend believed as much as I did that a real and deadly virus was spreading and there was an imminent risk of this affecting us all, including the Uber driver. Didn’t the Uber driver have an interest in knowing about it? Wasn’t it, if anything, our responsibility to tell the Uber driver? Is the concern that, as a normal person, the Uber driver is incompetent to manage himself, and will just scream and run around or buy poorly selected prepper equipment?

In conversations about AI risk, I sometimes see the same thing. Geoffrey Hinton says he thinks there’s a 10-20% chance of human extinction, and some people seem genuinely most concerned is that maybe the press didn’t add enough disclaimers about the process by which he reached that number, and the public may get unnecessarily worried. I agree it would be non-ideal if people were 20%-level worried when they would only endorse being 7% worried on further methodological inspection. But among non-ideal aspects of a situation where most of the relevant scientists believe their field is heading toward a modest-to-strong shot at killing us, it’s interesting to rate “maybe people will be too concerned” as a top concern.

What is going on? In this particular case, I could imagine behaving this way if the original communication seemed dishonest. But finding this dishonest seems similar to complaining if someone yells “fire!” that they should have yelled “I subjectively guess that it’s highly likely there’s a fire because of the smoke and flames but I’m not an expert”. And it seems like there’s something else going on with wanting that.

A related phenomenon: people casually mention ‘causing a panic’ as a thing that is assumed to be too terrible. Like, yes there’s some upside to warning people that there is a major threat to their lives that they can do something about. Maybe doing that will help stop the world from ending. But! What if they get all emotional? They may not act in the most clear-eyed and rational way. They may talk to each other in epistemically unvirtuous fashions and get even more concerned. They may buy too much toilet paper or run on a perfectly functional bank or protest for poorly designed policies.

I mean, indeed this is all worse than them addressing threats in the most rational and optimal way. But how is it a problem that even ranks compared to them not addressing threats because they don’t know about them? And who are you to not tell people about genuine risks to them that they would act on, to protect their feelings or because they would be more upset than you want?

I understand that gender is complicated and difficult, for the whole species (and honestly probably more so for some other species). And it can be hard to tell exactly if anyone is behaving badly regarding it, at least in my modern bubble. Maybe women just aren’t that into designing programming languages? Maybe the thing I’m saying is just boring and a man is saying a more interesting thing?

But a thing that is undeniable is that women want to open jars, dammit! What’s your nuanced explanation there, Bonne Maman? Does the proper amount of friction for maintaining spread safety fall just between the male and female human grip strength distributions?

This study suggests that would be about 400N Fmax (though this would not avert most elite female athletes acquiring jam, see second figure, and the pictured participants are young adults):

The distributions are really surprisingly not-overlapping!

90% of females produced less force than 95% of males. Though female athletes were significantly stronger (444 N) than their untrained female counterparts, this value corresponded to only the 25th percentile of the male subjects.

We know that men and women have different grip strengths! We know that about half of people are women! Why do so many containers require women asking a man for help in order to open them? (Or carrying around an opening tool or living in a kitchen?)

Yes, strength required to open packaging ranges across a wide distribution, but I note that very few are impossible for anyone to open, so it seems like some effort somewhere goes into keeping them in the feasible range, and that effort does not seem to care about it being reliably feasible for people like me. I don’t imagine Bonne Maman wants to stop women getting to their jam—I imagine that nobody cares.

I thought about this most when I lived in a group house with a shared bulk stash of Gatorade, and any time my woman housemate or I wanted to drink a red one we’d have to ask a guy to open it for us. But these days I also often hurt my hands opening (or failing to open) things, and while I’m sure I’m low in the female grip strength distribution—and may also be high on the ‘unreasonable anger about anything nearby when my hands are hurt’ distribution—I don’t think it’s just a me issue, and in the moment it always feels like a ‘fuck you, raspberry jam isn’t meant for you’.

Then—for me at least—there are different mental states that don’t immediately feel like anything, but where in acting I notice that my behavior is different, or different things feel easy or impossible. For example:

• If I’m in a lot of pain or distress alone, I might feel like I couldn’t compose myself. But in fact if company shows up, it becomes natural to pull myself together.

• If there’s a time limit, I will often find a vast well of motivation that was otherwise non-existent. The day before a deadline I will smoothly (if with a lot of effort) do ten times as much as on a normal day.

• On some days at least, if it’s 8pm, things will become possible that I could only have dreamed of at 11am.

• Riding my bike to the station, then taking it on the subway, then riding it again to get to a party seems like not a big deal with a friend, but like a prohibitive ordeal on my own.

Some of these are very important! For instance, in causing myself to get ten times as much work done sometimes, or to travel to places, or to not despair if things seem impossible at 11am.

But how many more shifts like this do I never notice because I don’t probe the range of relevant behavior in every possible circumstance? If different humans have similar mental architectures in this regard, can I get a list of the common ones somewhere?

Meanwhile award-winning journalist Ben Bradford of NPR has made a podcast version of my case for AI x-risk that I am thrilled with!

(Bonus within the 39 minutes: what Hamza Chaudhry of FLI thinks we should do about it—who I was delighted to later meet as a consequence!)

If you or anyone you know could do with a quick and gripping rundown of why this is a problem, try this one.

Get it on any podcasting app here: https://pod.link/1893359212

The NPR press release has more context on the rest of the series, assessing different possible sources of doom.

But sometimes games teach people less obvious things—things that are more experiential or ineffable, things that you didn’t know you didn’t know, concepts that stick in your mind, deep things. Here’s my list of games and their interesting real-world updates, as experienced by me or my friends:

Dominion: Don’t invest for eternity. When casually improving or protecting or investing in things, it’s easy for me to treat life (and perhaps even the present period) as basically eternal. In fact I shouldn’t, but it can take many years of living to really feel how likely it is that you’ll leave your perfectly wonderful house within two years, or just keep on aging. Dominion lets me feel that in a matter of hours, by tempting me to invest in a beautiful and effective deck that will do amazingly for the rest of eternity, then making the other player win by haphazardly buying a handful of provinces before I’m done. Which is very annoying, and I do hold against it.

**The Witness: **there is nothing in The Witness (at least near the start, I haven’t played it all) that you can pick up and take with you. No objects, no points, no manna, no health. It’s just you, walking around in a world. Something about that feels like it would be deeply unsatisfying—like what is a game, if you can’t get, y’know, things, dings? Part of me thinks that GETTING is equivalent to satisfaction, in spite of all the evidence to the contrary I keep pointing out to it. And The Witness is not where I came to realize that. What The Witness made me feel is that knowledge is a REAL thing you can GET, like an object. Not some hand-wavey second-rate bullshit thing that philosophers pretend to get off on. In The Witness, while your character walks around, impermeable to the world, you come to know more things. And knowing more things lets you go to places you couldn’t go to when you knew fewer things. The game on the computer concretely changes from you picking up knowledge, that ethereal thing in your mind. This is of course how everything is, but I suppose the absence of any other form of ‘picking up things’ in The Witness made me actually feel it.

**Minecraft: **How many of my difficulties in life are not this-life specific. How to live as a creature with different boundaries of personal-identity, e.g. the world spirit. Much more about these in my previous post, Mine-craft.

Return of the Obra Dinn: If at an event where lots of people are saying their name and what they do or something, I am usually bored and don’t expect to remember these things. Return of the Obra Dinn is a game where you have to figure out from minute clues the names and causes of death of a lot of characters. Once at a networking event, I decided to think of it as like a sequel of Return of the Obra Dinn—I could see all these people sitting around the table, and my quest was to pin a name and a deal to each of them, and this introductory section was currently showing me crucial information. I found that this was a very different mental state. So I suppose I learned that whatever I was normally doing in ‘trying to learn’ things about the other attendees, it is an extremely pale cousin of the curiosity I can feel in a different mental state, and that different mental state is actually fairly different, and naturally invoked in RotOD and not networking introductions.

**Dungeons and Dragons: **Caitlin Elizondo says DnD has given her a few concepts that make a difference to her thinking more generally. The concept of ‘will saves’ has given her more empathy for situations where someone wanted to but failed to do something. The six DnD stats helps her access the framework where there are different types of competency that are valuable for different tasks—obvious in theory, but easier to think in terms of with this structure.

**Poker: **the feeling of being ‘on tilt’

Boggle, Set, Ragnarock: the feeling of flow. Ragnarock is mine, and I would have said I’d experienced ‘flow’ elsewhere, but Ragnarock is sometimes more like an altered state than other such experiences I’ve had.

Civilization IV: I used to lose at a scenario then go back and play it again over and over changing things slightly until I won, which gave me a vivid sense of how suboptimal my native strategy is, presumably also in life. Which is obvious in theory, but it’s different to really feel how much better I would live this day if I was doing it the twentieth time with a laser focus on winning.

Games in general: the experience of addiction, sadly. I’ve always struggled to keep up habits of taking addictive substances, so I infer I’m unusually safe from chemical addictions (I used to play Civilization for five minutes as a reward if I remembered to take my amphetamines). Games are I think the thing I find most seriously addictive. Which has definite downsides, but it is certainly also an interesting experience that helps me understand the wider world better, and where I would be missing something if I just read about addiction in the abstract.

Do you have any to add?

[ETA May 1: I’m adding more I hear in the above list, and also see many good additions in the comments!]

Which leads me to wonder, supposing it’s true that many people are too deferential, what might people do to change it? And can I offer them useful advice, as a person who might be not deferential enough?

Tonight I talked to Joe Carlsmith about this; here are some ideas mostly from the conversation:

1. A thing that has discouraged me from having independent views and broadcasting them is the concern that my views are extremely ignorant. At the normal pace of new information appearing, I am just too slow a reader to be acceptably up on it. At the AI-news rate, it’s very hopeless. And even if you recognize that situation at a high level, it can be easy to get to thinking ‘I want to write something about W, but I’ll need to read X, Y and the sequence on Z and all the responses to it first’.

It was helpful to me to give up on this kind of expectation, and accept that I’m going to be ignorant and have views anyway. I think this is the right thing to do because a) everyone is fairly ignorant and we don’t want the public discussion to be only the few people who don’t realize they are ignorant or care, and b) saying what you guess is true then letting people point you to why you are wrong is often more efficient than scouring all writing on the topic, and c) there’s value from more independent thinking on a topic, and being informed comes with being less independent. Bringing us to—

1. Being sufficiently out of the loop can actually help, as long as you are bold enough not to be silenced by this—if you don’t know what others’ views are, you have to come up with your own.

2. Focus on having your own beliefs at a relatively high level. For instance, “Shouldn’t we be stopping AI though?.. Wait, does that argument make sense?” is the kind of thing you can think about and discuss reasonably well without needing to know a lot of technical or fast-moving details, until a more manageable few are brought up in the argument. And my sense is that these kinds of questions—e.g. is our basic strategy what it should be?—are actually neglected.

3. Which brings us to status. Intellectual deference probably follows normal patterns of status-based deference. So it probably helps to be either high status or arrogant. That’s a lot of effort, but you can have the experience of being high status or arrogant by talking to people who are relatively lower status or deferential, such as children.

4. It probably helps to be brought up in a situation where you learned to distrust the thinking of everyone around you. It’s probably ideal to be taught by your parents that everyone else around is an idiot, then to come to distrust your parents opinions also.

5. That’s hard to get later in life. But perhaps you can get something similar from experiencing apparently venerable intellects confidently asserting things, then later observing them to be false.

6. If you are in conversations where it seems like the other person isn’t making sense, try to assume that is what’s going on, rather than the potentially much more salient explanation that you are a fool.

7. Give esteem to people asking potentially silly questions. It can help to expose yourself to impressive people who do this.

Niels Bohr quotes are helpful (HT Wikiquote)

1. Refuse to ‘understand’ things unless they are very clear. I don’t really know how to do this, because I don’t know what the alternative is like—being steadfastly confused about things seems to come naturally to me and I don’t know how else to be, but maybe you have both affordances available here and could lean one way or the other.

2. Something something do real thinking versus fake thinking. Ironically, this point I am deferring on, because I haven’t finished listening to Joe’s (so far very interesting) post.

3. If you are going to pass on information that you don’t deeply understand, track that it is a different thing, for instance by saying ‘something something…’

Interviewer: It sounds like you’re pretty excited about self-driving cars. Weren’t you just saying that unemployment from AI is on some kind of very overlapping continuum with extinction from AI? Isn’t rooting for self-driving cars rooting for AI unemployment here, and thus extinction?

Katja: Hmm. Well first I should say, I’m actually fairly neutral on unemployment in general from technology. If technology makes it overall easier to produce what we want, but empowers some people over others, that change in power might be a downside (or not), but if so, it’s one I’m inclined to solve with direct redistribution rather than having the people who would be disempowered do unnecessary busywork to ‘earn’ their living.

Interviewer: Ok, so you think AI unemployment is different?

Katja: Yes, because it involves the disempowerment of humans in general in favor of non-people entities whose empowerment has a decent chance of spelling our ruin. Doing things the hard way to avoid that happening isn’t busywork, it’s very valuable. I don’t usually want to take sides between different humans systematically—society seems probably best served by letting the most effective production methods win out in most cases. But sometimes there are entities who produce things efficiently, and you still shouldn’t trade with them because it empowers them. It’s a lot like not trading with Nazis (broadly—I’m not saying AI entities are evil in the same way, just that their empowerment has a good chance of leading to genocide or omnicide).

Interviewer: Ok, but aren’t self-driving cars AI?

Katja: Yes, but the class of entities I don’t want to empower isn’t ‘AI’ really—it’s more like ‘AI agents’. Though also, the processes that are creating them, such as LLM companies, which complicates things. Self-driving cars are narrow and not much like entities that can be empowered. And my understanding is that we could have perfectly great self driving without using risky AI. But maybe I should be opposed—I’m not sure how to think about what class of entities I should not want empowered.

Interviewer: Are you just in love with self driving cars because they would be so personally convenient for you?

Katja: That is probably playing a role. A car with a random stranger in it is just so much less what I want most of the time than a car by myself. I like to imagine that Uber was invented like, “New startup idea: Chatroulette but you’re stuck in a moving vehicle with the person!” I would feel worse about the end of driving as a human profession if I felt like human drivers consistently did the job acceptably well. But the rate of drivers around here seeming chemically impaired or choosing to drive on the kerb of the freeway to get around other cars, etc, and also talking to me when I don’t want to talk, means there are a lot of cases where I would like to go somewhere in a car except it seems to awful so I don’t.

Interviewer: So how was the self driving car last night?

Katja: non-existent. Once I arrived at the airport, the Waymo app informed me that I wasn’t allowed to get cars there, because they are rolling it out slowly or something. I considered trying from one transit stop outside the airport, but since the available Waymo map was a very uninformative cartoon of the Bay Area and it was after midnight, that felt risky.

Interviewer: Did you give up?

Katja: Not immediately. I had also been told that people are taking ‘robotaxis’ all over, so I looked that up. I couldn’t immediately figure out what it was by Googling and looking on the Android app store, so I messaged some friends, and they directed me to an app called ‘RoboTaxi’ purportedly from Tesla but with barely readable and amateurish font and 57 reviews. As is often perplexingly the case with things of importance to a lot of people from very well known brands, I felt like I was exploring an obscure frontier that nobody had tried to use before. (You want to do what?? Get a ride in one of our cars?? And you want to do it through an app?? And you want to know where they are available??) I logged in and it told me the airport was also out of bounds. So then I gave up.

Interviewer: How did that make you feel?

Katja: ashamed

Interviewer: That makes sense. Why did you even so brazenly think you would probably be able to get a self-driving car from the airport?

Katja: well locally, because Waymo had a map indicating that the airport was within their zone, and I figured Tesla wouldn’t have such a can’t-do attitude. But more fundamentally, I guess I haven’t properly internalized how opposed airports are to efficient travel. Seeking a human-driven car after all this, I was reminded further because the location of the rideshare pickup at the airport and the signage indicating the location, both seem like they should probably be crimes.

Interviewer: Might it be an even broader problem with your level of techno-optimism? Weren’t you just the other day very disappointed by a futuristic kettle? Perhaps you need to learn that everything is shit?

Katja: maybe, but I don’t know, sometimes technology really changes things. I remember before Uber, when I just had to phone a person at a taxi company and ask them to come and collect me and then wait for an unknown period of time, and worst case give up and walk home.

Interviewer: How was your ride home last night?

Katja: Pretty good. The Lyft driver didn’t perceive my initial desire not to talk, and so we had a detailed discussion of Yemen, his life as an immigrant, his family, arranged marriage, romance in Islam, experiences running different businesses, the nature of business partnership, AI risk, and other drivers’ views on automation. We exchanged details so we could interact again.

Interviewer: Do you really think your quest to instead drive home in sterility wasn’t completely misguided?

Katja: Humans are great, but you have to be allowed to want solitude sometimes. It follows that you should probably be allowed to want solitude while also getting to another location. That said, I probably want solitude unhealthily much, and underrate the loss of human connection from these innovations. Maybe there should be a tax or something.

In my own life at least, their actual arrival has been gradual and underwhelming. At some point I learned that you could call Waymos to drive you around a subset of the city of San Francisco. This was wild and exciting, but not actually very useful, since getting anywhere in San Francisco seemed to require traversing a greater subset of it than that, plus I nearly always want to also drive to or from Berkeley. I took Waymos a couple of times, and one of them might have even had transport value.

In the last few days, multiple people not from the Bay Area have casually assumed that I take self-driving taxis all the time, which caused me to investigate and learn that Waymo is now all the way up and down the West Bay Area, seemingly including the airport. So tonight when my plane lands I hope to finally get half way home without a driver! It’s not clear this will improve my experience, since I will then have to catch a Lyft in downtown San Francisco, and probably explain to them that I have all this luggage because I took a self driving car as far as I possibly could before resorting to a human driver. But I am still excited.

Some people are extremely concerned about extinction and perhaps even indifferent to total unemployment. Some people think of moderate AI unemployment as a realistic and concerning issue, and AI extinction as science fiction.

I think of AI unemployment and AI extinction risk as basically the same issue, and in likely scenarios, happening together.

At a very high level, I’d say the argument for human extinction from advanced AI is something like this:

1. We’re going to make AI that can do everything better than humans

2. We’re going to make that AI into agents that navigate the world independently and do what they want

3. We are not going to make those AI agents want the right things

The basic issue is that in the presence of more capable agents with different goals, humans are less able to get resources and influence, and direct them toward the humans’ goals.

One way ‘losing power to more competent agents’ could look is that a surpassingly smart AI agent intentionally eradicates humanity. But killing everyone and controlling the world is a pretty wild corner case of ‘using your competence to control the situation toward your own preferences’. In particular, it has never been seen before, though the new AI situation might make it possible.

The traditional ways that humans make use of competence to influence the world include earning salaries then spending the money on things they want, earning investment income, making and using alliances, persuasion, taking political actions, etc.

If no ultrapowerful AI appears and exterminates us, I think we have every reason to expect ruin from AI sapping our power and resources by these more traditional methods. Outcompeting us as labor, outcompeting us as informed capital holders, outclassing us at political strategy and persuasion, and controlling the conversation.

It’s true that if humans were only excluded from the employment path to resources or influence, this would merely be an excruciating upheaval on a massive scale, and probably not herald extinction.

But unemployment here is just the most legible tip of a sprawling shitberg. It’s just not plausible that humans are unemployable, but they are doing well at political strategy and persuasive communication. Unemployment goes with losing power across the board, except insofar as power is granted by whoever has it by virtue of might. That is, insofar as AI cares about empowering us.

So unemployment could happen without extinction (if we successfully built AI that cared about us in the right way) and extinction could happen without unemployment (e.g. if an extremely competent AI system decides to exterminate us). But in a lot of cases they not only coincide, but are the same issue.

Asking if someone is more concerned about unemployment or extinction is like asking if someone primarily wears a seatbelt when driving to avoid having their body flung through the windscreen, or to avoid dying.

If powerful AI agents have their own agendas, those agendas will win out. They might win out in one crushing step, or win out in a trillion small familiar ways.

The next day I saw Jonathan Groff perform at the start of a TIME 100 event. He did feel like another human right there in the room, but still somehow surprisingly distant. His song didn’t touch me, though my previous experience delighting in his presence on YouTube led me to have other expectations. And his song was Sondheim, which should also support other expectations. Toward the end he said something about—I think—his work being about connection with us all and being together, and that explicit thought reached me better than anything in the music.

Having enjoyed the first Broadway show of my life the day before, and still being in New York, that evening we went to see another one: Every Brilliant Thing. It was a one-man show currently starring Daniel Radcliffe, and involved a lot of Daniel running on and off the stage and borrowing objects and asking audience members to briefly represent his high school teacher or vet or crush. It was also quite poignant. So if any performance was going to feel like a real person was in the same room as me, and like that person was reaching me emotionally, this would seem to have a good shot. And Daniel did seem like a real person over there. But I recognized the movingness of the story more than actually being moved.

The play was about a long list of things in life that are ‘brilliant’, which I hear as both ‘great’ and as ‘brightening’. The list made me feel uneasy, because I recognized the things as good, but I didn’t feel it. Ice cream, sure. The smell of an old book, okay. Bed, yeah I guess I like it quite a lot more than not-bed. This was all somewhat fitting with the themes of the play, in which the brilliance of things was recognized to different degrees by different characters at different times and levels of depression. But it still made me feel improper and distant from other people: the audience was meant to understand these things as brilliant. I cried a little about that afterwards.

But it also reminded me later of an obvious-but-hard-to-enact thought: if you don’t feel the goodness of things much at some point, it’s not an indication that everything has gone wrong, or that the world is no longer good, or never was, or that there is something badly wrong with you. This is clear in writing, but if I don’t explicitly consider the issue, it is easy to interpret such experiences of nothing seeming good as some variant of ‘nothing is good’. Which is about as correct as having a numb foot and describing the situation as ‘there is no floor’. The picture in my mind is of a sailor in darkness: sometimes you can’t see the stars, and then you have to navigate by memory. And the darkness is about your current location, not the world—it’s possible to get to other non-dark places, even if you can’t see them from here.

One of the last brilliant things involved the sound of a record crackling into the start of a song, which I was surprised to learn that I did feel something about. Perhaps because it reminds me of being alone, in a nice library, with pen and paper. That kind of state quite close to the sublime! Which reminds me that things can be sublime.

These forays into group emotion left me feeling somewhat like a distant and unmoved observer, but an earlier example is also interesting. In our first hours in town, we rushed out to see a live comedy show. It was in a crowded underground bar, and we were in flip-out seats in a walkway. I was struck by how much even the emcee made us laugh—they were funny, but I bet we laughed more than watching the most celebrated comedians on a screen. And it seemed like a fuller laughing experience—laughing with group, together. I hadn’t really thought of comedy as much better live. I wonder if in-person social orchestration is a big part of making people laugh, beyond the performance-as-separable-pixels-and-sounds. For this show at least, I was easily brought along with the emotions intended, and felt closer.

Is it less vulnerable to go along with comedic emotions than be moved by serious ones? And with a crowd I feel self-conscious? I wonder if my relationship with the other people in the room matters more than the performer.

Sitting outside a restaurant later, watching people and cars pass by, I thought about how sometimes everything seems brilliant to me, and remembering that made everything seem a bit brilliant.