In April, the AI Futures Project came out with AI 2027; I read it one night in May 2025. Here I wanted to write down observations on the piece: some more low-level takeaways. I’m not going to synthesize things into an assessment or opinion, but I hope that even the more dry format carries a sense of things that I personally found interesting.
AI 2027 is a text, a story in its form. So we can do a “close reading” of the text. I also enjoyed clicking through several of their external links to papers and blog posts. Those choices of what to link also form part of the text, and introduced me to new ideas in the conversation around AI safety, so I’ll be reflecting on those as well.
Observations:
They explain that a few of them came up with the policy predictions, but they got an experienced writer (Scott Alexander) to help them make the prose interesting.
At least part of the incentive of these authors is to get us (society) to think about and avoid these doomsday situations. That doesn’t need to imply a dishonest forecast, but I think can be taken into account when trying to understand the text.
I identified 3 categories of existential threat to humanity from AI that were portrayed in the piece. (1) Enabling new biological / chemical weapons. (2) Hacking into national government data and computer systems. (3) An AI self-improvement loop untethered from / acting against human control. For 1, examples given are “plans for synthesizing and releasing a mirror life organism which would probably destroy the biosphere” and “in mid-2030, the AI releases a dozen quiet-spreading biological weapons in major cities, lets them silently infect almost everyone, then triggers them with a chemical spray”. My perception is that these 3 risks have been consistently talked about in the AI safety community for some time, and it’s worthwhile to keep them front of mind. For an example of how a company factors these risks in (whether appropriately or inadequately is up to you), I found OpenAI’s preparedness framework (blog post, document PDF), which got an update in April 2025.
One perspective that I thought was communicated really well in the piece was that autonomous AI agents could take off rapidly and change the world within a few months. That transformation could come like a “thief in the night”, to use a biblical phrase. This message was conveyed through: (1) The start of the piece being quite boring and undramatic, as AI capabilities improve only marginally for all of 2025. (2) When AI capabilities do take off, it’s known to the top company building the AI and to the US / Chinese government, but not the general public. I think this is the most unique contribution of the piece, and it requires this long-form prose format to really sell.
In the piece, AI agents end up accelerating the rate of AI progress at the top company, because research / coding is done at a much more productive pace. “This massive superhuman labor force speeds up OpenBrain’s overall rate of algorithmic progress by “only” 4x due to bottlenecks and diminishing returns to coding labor.” An open question for me is, how much is the desire to accelerate companies’ own productivity a motivation for all the investment in AI coding assistants? I had always thought companies are making AI coding agents because it’s a familiar domain to them, and as my friend Andrew Liu pointed out, is a good business case.
One technical detail that seemed to really interest the authors is the idea of “neuralese” representations and “neuralese chain-of-thought” reasoning. In my words, the current best-performing LLMs generate sentences of English text as part of their internal reasoning process. This is called a chain of thought. So if you asked for 5 to the power of 5, or some other math problem, models that write out their scratch work tend to do better. Right now, chain-of-thought reasoning happens in English, but the authors posit a situation where models do this reasoning using thought sequences that are numerical (vectors) rather than verbal. They cite this paper as an example of what they have in mind. Although “To our knowledge, leading AI companies such as Meta, Google DeepMind, OpenAI, and Anthropic have not yet actually implemented this idea in their frontier models”, they write, “We are forecasting that by April 2027 the cost-benefit tradeoff looks much better for neuralese, due to developing better techniques and a larger fraction of the training being post-training.” This switch to neuralese reasoning is a prerequisite to how their scenario plays out, because it enables the AI agents to plan and communicate in secret without the humans being able to “read their thoughts”. In their slowdown situation where humans manage to wrest control over the AIs, “The agenda that gets the most resources is faithful chain of thought: force individual AI systems to “think in English” like the AIs of 2025, and don’t optimize the “thoughts” to look nice.” (I found the latter of these links, from the OpenAI blog in March 2025, to be a good read.) To me, this theme was where the forecast sounded most like an opinion piece: their single biggest recommendation seems to be to maintain English chain-of-thought reasoning in LLMs. In other words, don’t adopt neuralese reasoning! On the other hand, it seems like the fact that the authors had to “obsoletify” current chain-of-thought state-of-the-art might be a source of optimism with how things currently stand. I’m sure this has been discussed in many places, but you can read more by searching for “gifts” in this blog post by my friend Alex Irpan. From AI 2027: if “the AIs that first automate AI R&D will still be thinking in mostly-faithful English chains of thought … that’ll make misalignments much easier to notice, and overall our story would be importantly different and more optimistic.”
In one of the footnotes, the authors quote a 2017 email from Ilya Sutskever to Elon Musk, Sam Altman, Greg Brockman, Sam Teller, and Shivon Zilis: “The goal of OpenAI is to make the future good and to avoid an AGI dictatorship. You [Elon] are concerned that Demis [Hassabis] could create an AGI dictatorship. So do we [Greg and Ilya]. So it is a bad idea to create a structure where you could become a dictator if you chose to, especially given that we can create some other structure that avoids this possibility.” Here’s the link that the AI 2027 authors included, though I’d recommend starting here, which has a few select quotes as well as the original post. Personally, I gained a lot of respect for Ilya from reading his share of the emails in the record.