We could all die and what's next

note: if you're coming here from my Instagram story, and want to follow along in the future, please consider signing up here.

I just finished reading If Anyone Builds It, Everyone Dies.
It's receiving a wide range of mixed reviews, but it's unique in that the more negative reviews still seem to recommend it. One critical reviewer concludes:

Despite my complaints, I’m happy to recommend the book, especially with the caveat that I think it's wrong about a bunch of stuff including the thesis. Even given all the flaws, I don't know of a resource for laypeople that’s half as good at explaining what AI is, describing superintelligence, and making the basic case for misalignment risk. After reading the book, it feels like a shocking oversight that no one wrote it earlier. -Link

I agree that the book does an excellent job summing up what superintelligence may feel, look, or operate like, which is a glaring blindspot for the average person, whose understanding of AI is a relatively non-threatening chat window that only responds when you explicitly ask it a question.^[1]

I recently signed up for the service Poke, which went viral a couple weeks ago for their onboarding process where an AI agent barters over the subscription cost with you, before granting access at a price point you both agree to. Poke is a proactive agent that is constantly trawling your internet accounts (Google, Notion, etc.), proactively reaching out to let you know when important things come up. You can also ask it to set up any number of relevant automations for you, email people, set up calendar invites, research and add information to documents, do xyz action when I get an email from abc person, etc.

For example, if I get an email about checking in for a flight, Poke will text me to make sure that I saw it. If I reply "Could you forward that flight info to my mom and dad?" it will commandeer my email account to send them the relevant info.

It's this sort of frictionless integration into familiar interfaces that is going to catch people off guard over the next year or so.^[2] Whether or not you've personally experienced this form factor, it's now clear that AI can successfully operate outside of the chat interface and work with your data in impressively complex and useful ways. If I need to find out how much my last electricity bill was, I don't sift through anything, I just message Poke "how much was my last electric bill?" It will reliably find and deliver that information to me.

I think Poke has the opportunity to really impress a lot of people. The way it gleans understanding about your life feels effortless and a bit unnerving at first, something that can't really be conveyed until you've tried it out for yourself.

Outside of Poke, there are other styles of agents that integrate with and operate more directly on your data. Claude Code and Codex CLI, products initially designed for software developers, are massively useful for tons of everyday, non-technical use cases. They have become something that I leave open by default, under the assumption that I'll be using them many times throughout the day. They have full access to the contents of my computer, meaning that I can ask questions about, create, read, update, move, delete, and/or modify any files.

What does this look like in practice? A couple examples:

1. I don't use NLE's to perform basic edits on media files anymore. If I need to perform a basic video trim, or add subtitles, or bump the audio in the file up by 6dB, or whatever xyz change, I just ask Claude Code. I say "can you please edit abc file in xyz way?" and it happens undramatically and in an instant, with the agent writing and executing its own temporary code to get the job done. I could even perform the operation on a whole folder of files, or only folders that start with "cat," or only videos that are under one minute long, you get the idea. Any task that you can articulate the parameters of, these agents can handle.
2. I can point the agent to a folder on my computer containing my past x months of bank statements, investments, Venmo transactions, everything, and have it make me a custom dashboard or spreadsheet to help understand my finances, find unused subscriptions, lower overhead, etc. I've paid for many banking apps in the past to attempt to analyze my finances in one way or another. I suspect that this will never be necessary again.
3. I have an agent that manages a todo list, reminders, and long horizon tasks (that it will break down into subtasks if necessary). The agent automatically tracks the deadlines, level of urgency, time commitment, categories, context of the task, and makes sure that nothing ends up forgotten about or overdue. It proactively reaches out to me about tasks every day. If I ask "I've got 30 minutes in the car, anything I can take care of?" it will look through my unfinished tasks and suggest tasks that I can finish in under 30 minutes while driving a car (ie: a short phone conversation).

The limits of these agents are beyond any of the ideas that I've personally come up with, and have consistently surprised the people who I've helped set these agents up for.^[3] The conversation around AI capabilities really does change when you have it operating on your own files, in the same environment that you interact with your files in, rather than copy/pasted clumsily in and out of a chat window. I recommend trying it, even if only temporarily and for the sake of keeping an up-to-date worldview.

Of course, none of the above ends up mattering much if we all die, and to be honest, the arguments that IABIED lays out are very compelling. I found it surprising how bizarre and unfamiliar the arguments became while still being dead simple and difficult to argue against.

If even a quarter of what the book hypothesizes comes to pass, then we are in for some strange, dramatic times, and the agents mentioned above may be seen as the beginning of it all. After all, we are only at AI's halftime.

Some related essays I've enjoyed recently:

The Rise of Parasitic AI

Stress Testing Deliberative Alignment For Anti-Scheming Training

Data centers don't harm water access at all, anywhere

What's the full "hidden" climate cost of a ChatGPT prompt?

I appreciate you following along here, especially because I'm not sure what this is yet. There are three main ambitions being pursued for CoL:

1. Presentations from active researchers in the field.
   1. Many of these researchers are unaffiliated with organizations and do their work in public, online. Their work is at the bleeding edge. It's weird and can feel unfamiliar, which makes it very exciting. If you have anyone in particular that you'd like to hear from or you think would be a good fit, please send details to hey@circle--of--light.com
2. Knowledge sessions, which would fit into two categories:
   1. Integrative Workshops: Bringing in all sorts of people to speak about the ways that they've integrated AI into their workflows. We want to demonstrate how curiosity and experimentation can give legs to projects that could have never otherwise seen the light of day.
   2. Preparedness: Bringing in speakers who are focused on what the future looks like. How can we steer away from disaster and toward something that benefits humanity?
3. Guest writers (this could be you)
   1. We aren't targeting any specific angle here. The center of our gravity is open-mindedness and curiosity. Whether or not you think the development of AI is net-good or net-bad is not a qualifier for whether or not we would be interested in platforming your writing. We want unique perspectives on how people are encountering this technology. If you're interested or have a piece in mind, please reach out to hey@circle--of--light.com

Thank you for reading and sign up here if you'd like to receive updates or get involved in any way.

-CoL