Noam Brown – Agent swarms, alignment, & recursive self-improvement
发布时间 来源
Episode 设置
摘要
New episode with Noam Brown.We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.And we also discuss how we will know if the models are actually aligned before we kick off RSI.Watch on YouTube; read the transcript.Sponsors* Jane Street has been interested in AI for a lot longer than you’d think, and not just for trading. In 2011, a full year before AlexNet and over a decade before ChatGPT launched, they hosted the first FOOM Debate between Eliezer Yudkowsky and Robin Hanson on whether AI would lead to an intelligence explosion. Now Jane Street is revisiting the question with a new panel: Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla, hosted by Ron Minsky in San Francisco this October. I expect it to be a truly excellent conversation. Register at janestreet.com/dwarkesh* Grok Bot has made handing off work super easy. It runs on its own cloud computer, where it installs the tools it needs to handle tasks end-to-end. For the podcast, we use Grok Bot to help produce our videos. You may have noticed that our ads feature animations of real websites. Getting these pixel-perfect used to mean running a convoluted, multi-step workflow ourselves. Now we just let Grok Bot handle it. Best of all, Grok Bot has learned all of our specs and preferences, so we don’t have to redescribe the task each time! Try Grok Bot for yourself at x.ai/bot* Antithesis gives you the confidence of a giant test suite without actually having to write one. Say you’re doing a major backend refactor: building enough tests to trust it could take weeks. Antithesis solves this by running your software through countless simulated worlds, injecting faults and hunting for failures. On any PR, you can turn a dial to decide exactly how much testing you want. And because every run is fully deterministic, agents can branch off the moment a bug appears, rewind it, inspect memory, and replay it, all while the original test keeps running. Learn more at antithesis.com/dwarkeshTimestamps(00:00:00) – Multi-agent and Navier-Stokes(00:15:28) – How will AI firms work?(00:22:02) – What math progress tells us about recursive self improvement(00:40:22) – Hugging Face and alignment(01:01:18) – The internal/external model gap(01:08:34) – Chain of thought is degrading(01:14:12) – How will we know when alignment is solved? This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com
GPT-4正在为你翻译摘要中......
中英文字稿 
今天，我正在与Noam Brown聊天，他是OpenAI的研究员。他是O1和推理模型的基础贡献者之一。现在，他正在研究多智能体系统。谈到这个，你们上周宣布用一个由10,000个不同AI代理组成的系统解决了一个百万美元的问题，这个系统在88小时内使用了1300亿个令牌。我想与你交谈的原因之一是，我认为大约两三年前你是第一批考虑如何通过推理模型预见未来的人之一，因为如果你扩大推理计算能力，你可以预测模型在几年的基础功能。
▶ 英文原文 ⏱
Today, I'm chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became O1 and the reasoning models. And now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the million price problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours. One of the reasons I'm interested in talking to you is I think you were in the first people maybe two or three years ago who was thinking about how the reasoning models would allow us to see into the future because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future.
我觉得你现在处于一个类似的位置，可以帮助我们理解未来的能力会是什么样的，特别是考虑到我们现在可以实现代理规模的巨大扩展。当我思考这个问题时，如果你把这些推理模型的性能放在横轴上，测试时间的计算量放在纵轴上进行比较，基本上在任何推理基准上，你都会发现一个非常清晰的模式：这些模型花费更多时间思考答案时，它们表现得更好。这其实很自然，这和人类是一样的。如果你在参加SAT考试，而只有五分钟的时间来完成整个考试，你可能不会考得很好。但如果你有五个小时，你可能会考得更好很多。
▶ 英文原文 ⏱
And I feel like you're in a similar position now to help us understand what future capabilities will look like given the enormous scaling of agent sizes that we can do right now. So the way I think about it, when you plot the performance of these reasoning models with test time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's the same thing with people. If you're taking the SATs, you have five minutes to go through the entire exam, you're not going to do very well. If you have five hours, you're probably going to do a lot better.
这些人工智能模型非常相似，它们会用这段时间自言自语，思考问题，分析不同的案例，排除不同的可能性，并在之前的发现基础上进行拓展。问题在于，当你推动这一过程越来越深入时，会遇到延迟瓶颈。你不会想等上三年才得到一个回应。所以，很多人采取的方法是进行并行化。他们会组建一支团队。如果你要创立一家公司，你会想要聚集一群人，这样可以加快速度。对于这些人工智能模型来说也是一样，让多个智能体同时工作会有帮助，因为这样可以更快地完成任务。
▶ 英文原文 ⏱
The AI models are pretty similar and they'll spend that time doing this monologue to themselves, figuring out, going through different cases, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don't want to sit around for three years waiting for a response. And so what you can do is what a lot of people do is they paralyze. They just get a team of people. If you're going to found a company, you want to get a group of people together so you can go faster. It's the same thing with these AI models that it helps to just have multiple agents working on something because they can just go faster.
多代理系统是一种以并行而非纯串行方式扩展测试时计算的方法。虽然效率较低，因为单个代理无法完全掌握所有背景信息，但如果操作得当，它仍然是扩展测试时计算的非常有效的方式。好了，我要问一些简单的问题，因为这些系统是未发布的模型，所以我们尚未公开看到这些系统的运作方式。因此，我对这种系统的性质有很多困惑的地方。
▶ 英文原文 ⏱
And so multi-agent is a way of scaling test-time compute in parallel instead of purely serial. And it is like less efficient because it doesn't have, it's not like a single agent has all the context to itself, but it is like a very effective way of scaling test-time compute if it's done well. Okay. I'm going to ask a bunch of naive questions because these systems, so this is, this is an unreleased model. So we haven't publicly seen how these systems work. And so I just have a bunch of ways in which I'm like confused about like what the qualitative properties of such systems are.
我对你在短时间内能够集中如此大量的认知精力感到震惊。如果用1300亿个标记来形容，这相当于一个人像全职工作那样，不间断地思考，就像从古代苏美尔一路思考到今天，一个人连续不断地思考需要大约4000年，每周工作40小时，每天工作8小时。而你却在88个小时内就完成了。我认为，从质量上来说，这是一个非常重要的考虑因素。
▶ 英文原文 ⏱
I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as, you know, full-time job, stretch back to back, 130 billion tokens would be a human thinking for like 4,000 years, you know, eight hours a day or something, working a normal work week. So starting from like ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours. I feel like qualitatively that is a super important consideration.
我很惊讶，在这么大规模的协作中，平行化的惩罚并不更大。你可以让一万个代理一起合作。也许是因为这些代理在协作方面比人类更擅长，他们的速度要快得多，以至于能够在这么大规模下高效协作。或者，也许我不知道，也许存在一项巨大的平行化惩罚。好了，我们来谈谈平行化惩罚吧，然后我们再讨论一些定性的东西。因为事实上，我们在多代理扩展到这种规模方面的科学研究并不够完善。
▶ 英文原文 ⏱
And I'm surprised that there isn't a bigger parallelization penalty, that you can just have 10,000 agents collaborate. And because maybe the agents are better at collaborating than humans might be, they're going much faster that they can actually productively collaborate at such a big scale. Or maybe they, I don't know, maybe there is a big parallelization penalty. Yeah. Let's talk about the parallelization penalty. Then we can talk about the qualitative stuff. Because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale.
好的。所以当我们发布5.6版本时，我认为这是我们第一次在模型中引入了真正的多代理系统。实际上，我们在博客文章中确实展示了一些多代理系统在扩展性能方面的图表。因为我们将此作为一个选项，它被称为超模式。默认设置是四个代理，但你可以将其设置得更高。在图表中，我们展示了在一些测试基准上，一个代理的性能、四个代理共同工作的性能以及十六个代理共同工作的性能。
▶ 英文原文 ⏱
Yeah. So when we released 5.6, I think that was the first time that we had a proper multi-agent system in our models. And we actually did in the blog post show some plots of the scaling performance of multi-agent systems. Because we have it as an option. It's ultra mode. And the default is four agents, but you can set that to higher. And in the plot, we show, okay, here's what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together.
你会发现，对于某些基准测试来说，这取决于具体的基准测试，如果有四个代理一起处理问题，问题的解决速度会加快一倍。基本上，你需要支付两倍的费用，因为有四个代理在做同样的工作，并且所用时间减半。如果你增加到16个代理，情况类似，但效率稍微降低一些，不过你仍然能看到性能的提升。这是线性的时间加速还是次线性的加速呢？我认为是次线性的，但这依赖于具体问题。例如，数学问题相对容易并行处理，虽然不是最容易并行的，但确实很适合这样处理。
▶ 英文原文 ⏱
And what you see, and it depends on the benchmark, but for some of the benchmarks, basically if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern. It's like a little less efficient, but you continue to see that performance. Is it a linear serial time speed up or a sublinear speed up as you increase the number of parallel agents? I would say it's slightly sublinear, though it does depend a lot on the problem. So math, for example, is quite paralyzable. It's not the most paralyzable thing, but it is very paralyzable.
我认为网络搜索，比如进行深入研究报告时，我们需要查阅大量来源的这种任务，是很容易并行化的。我猜想像写小说这样的任务就很难并行化，所以可能不会因为有1万名代理一起写小说而有很大收益。就像你不会因为1万人一起写同一本小说而有很大收益一样。因此，性能取决于具体领域。我们在已发布的博客文章中测量过大约16个代理的性能。问题在于，将科学推进到1万个代理的规模非常困难，因为成本实在太高了。
▶ 英文原文 ⏱
I think web search, things like doing a deep research report, we have to look through a bunch of sources that's extremely paralyzable. I suspect that something like writing a novel would be very unparalyzable, so you would probably not see a big benefit from having 10,000 agents working on a novel together. In the same way that you'd probably not have a big benefit from having 10,000 people work on a novel together. So the performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is it's very hard to push that science to 10,000 agents because it's just so expensive.
你们只用一个周末就完成了这件事。但这只是一个数据点。我们不知道一个单独的智能体需要多长时间才能解决小说家笑话的问题，因为我们还没有进行过这个实验。也许我们会做这个实验，但就像我说的，那也仅仅是一个数据点。如果我们想要进行彻底的消融实验，那么实验的成本太高，无法扩展到那个规模。因此，我们需要采用一些系统的方法来研究当智能体数量增加到64、128、256个时会发生什么，从而了解它们的行为。但要将数量推到1万个，确定使用1万个代理与1千个相比到底有什么好处将会非常困难。
▶ 英文原文 ⏱
You guys just did it over like a weekend. But that's one data point. We don't know how long it would take a single agent to solve novelist jokes because we haven't done that experiment yet. And maybe we will, but I mean, that's also only one data point, right? And if we want to do a thorough ablation, it's actually, the experiments are just too expensive to go to that scale. So we have to do some kind of methodical science about what happens when you go to 64, 128, 256 or something and get a sense of the behavior. But it's going to be very hard to push that all the way to like 10,000 and know for sure what was the benefit that we actually got for using 10,000 agents versus 1,000.
好的，我想强调一点，就是获得千禧奖问题的努力，并不是多智能体的功劳。我甚至不会把10%的功劳归于多智能体。实际上，我们训练了一个非常强大的模型，OpenAI训练了一个这样的模型，并且我们可以让这个模型在很长的时间跨度内运作，我们可以让它并行思考。但归根结底，我们之所以能够做到这些，是因为我们拥有一个通用的、非常强大的模型。
▶ 英文原文 ⏱
Yeah. One thing I want to make clear is that the effort to get, it's not like, the effort to get a millennium prize problem, this was not due to multi-agent. I wouldn't even like attribute like 10% of the credits to multi-agent. Like the reality is we've trained to, OpenAI has trained like a very powerful model. And we can get that model to operate over very long horizons. We can get it to think in parallel. But at its core, the reason why we're able to do this is because we just have a general purpose, very strong model.
我认为类似多代理系统的技术看起来很新颖，也很吸引眼球，这可能使它们得到了不成比例的关注。不过，核心原因在于这些系统本身是非常强大的模型。它们的泛化能力让我大为震惊。我指的是美国的这些系统，我不清楚这些系统是如何训练的，但可以推测它们是通过某种强化学习（RL）训练的。通常是通过一系列可检验的合成问题进行训练，与它们进行大量的强化学习。在整个训练过程中，我猜测模型并没有解决像“千禧年大奖问题”那样雄心勃勃的难题。
▶ 英文原文 ⏱
And I think things like multi-agents are flashy and are new. And that probably gets disproportionate credit for that reason. But the core reason is like, this is just a very powerful model. So the generalization is actually quite shocking to me. The U.S. systems, I mean, we don't, I don't know how these systems were trained, but presumably they were trained how RL training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process, I'm guessing, was like the model solving anything as ambitious as a millennium prize problem.
这个概念的推广非常有力，以至于一些更容易验证的问题也可以推广成很复杂的问题。我认为这是对的。首先，我们确实在非常困难的问题上训练模型，所以显然存在一个差距。比如，当我们训练某些类型的任务时，模型往往能完成比这些任务更复杂的目标。这其中的一个有趣挑战是，随着模型变得越来越智能，我们能向它们提出的许多问题反而变得太简单。
▶ 英文原文 ⏱
But the generalization was strong enough that like you could have this like much easier verifiable problems generalized to this much parallel effort on such a hard problem. I think that it's true. Like, first of all, we do train the model on very hard problems. So there is definitely a gap. Like we see if we train on some kinds of tasks, like it's able to do tasks that are more ambitious than that. There is an interesting challenge that as the models become smarter and smarter, the kinds of questions we can ask them, just a lot of them are too easy.
很难挑战这个模型。我认为这是一个有趣的现象。如果让我解释为什么你可能不会看到像大型语言模型这样的人工智能走向与AlphaGo、AlphaZero以及其他这些游戏AI相同的发展路径，原因之一可能是这种问题。在像AlphaZero这样的系统中，通过自我对弈可以获得无限的学习内容，因为你总是与一个实力相当的AI对战。
▶ 英文原文 ⏱
And it's hard to challenge the model. And I do think that that's going to be an interesting, like if I had to make an argument for why you might not see AIs like LLMs go the same path as AlphaGo and AlphaZero and all these kinds of like game playing AIs, it might be that this, this kind of problem that in things like AlphaZero, where you have self-play, you have an infinite curriculum. You're always playing against an AI that's like equally strong.
对于使用强化学习训练大型语言模型（LLM）这样的任务，目前的方式是给模型一个问题，让它去解决。如果问题太简单，简单到我自己一秒钟就能解决，那模型就没有真正学到东西。所以，如果我们没有足够具有挑战性的问题让模型去解决，那么这可能会导致进步变得更加困难。
▶ 英文原文 ⏱
Whereas for things like training in LLM with reinforcement learning, at least the ways that are out there right now, you give the model a problem and you ask it to solve it. And if the problem is so easy that I could just solve it in a second, it's not really learning anything. Yeah. So if we run out of problems to ask it that challenge it, then that is a plausible scenario where actually like, okay, it becomes much harder to make progress.
是的。我确实认为有办法解决这个问题。所以，我们还没有真正把它当作一个难以逾越的障碍来面对。我想，如果这成为一个严重的问题，还是会有解决的方法，但这确实是一个可能的情景。为了让大家更明白，我们在提到像AlphaGo或AlphaZero这样的程序时，我们指的是它们在达到人类水平之后相对快速地获得超人表现。就像在围棋这样的游戏中，它们在大约一年的时间里，从击败欧洲国际象棋冠军，可能是世界排名第50左右，到击败世界冠军，最后变得比任何活着的人类强大无数倍。
▶ 英文原文 ⏱
Yeah. Now I do think there are ways around that. And, um, so we haven't really hit that, uh, as a wall yet. Um, and I think that if it ever became a serious problem, there would be ways around it, but it is like a plausible scenario. Yeah. Um, and just for the audience, when, when you're referring to like AlphaGo or AlphaZero, you're saying like getting superhuman relatively fast after achieving human level performance. Yeah. I mean, if you look at the trajectory of game playing as like Go, they, within a span of like a year, went from beating a European chess champion, being like, I don't know, like number 50 in the world to beating the world champion to being unimaginably orders of magnitude stronger than any human alive.
好的。那么有可能像数学这样的领域会有类似的发展轨迹，但我认为也有一个非常可能的情况是这种情况不会发生。是的。所以我想了解一下如果在六个月内人们能接触到多智能体系统，我们应该如何模拟与多智能体系统合作或雇佣多智能体系统的情形？我应该先谈谈这些多智能体系统究竟是如何运行的，我认为这与许多其他人工智能中的多智能体系统有很大的不同。
▶ 英文原文 ⏱
Yeah. So it's possible that domains like math, we see a similar trajectory, but I think there is a very plausible scenario where actually that doesn't happen. Yeah. Yeah. So I want to understand if in six months people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system? Yeah. I should start by talking about how these multi-agent systems actually work, which is, I think a very different way than a lot of multi-agent systems in other AIs.
很多人在研究用于大型语言模型（LLM）的多代理时，通常会采用这种非常框架化的方法。比如，可能会有一个协调者代理，把工作分配给多个子代理，并给予它们具体的任务。子代理完成任务后，将结果反馈回来。这种设置看起来很合理，确实有帮助。但是，这种框架也存在一些限制。例如，在这种设置中，如果协调者负责将任务发送给子代理，而子代理负责完成任务并返回结果，那么就有可能存在诸多限制。
▶ 英文原文 ⏱
So a lot of people that have approached multi-agents for things like LLMs tend to take this like very scaffolded approach where, you know, for example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task and the children work on it and then return their answer. And this seems like a very sensible setup, very sensible scaffold. It definitely helps. But there are a bunch of limitations with these kinds of setups. So for example, if in this setup you have a coordinator that's sending tasks to children, the children work on it and then return their answers.
好的，那如果给两个孩子布置类似的任务，会发生什么呢？他们可以互相交流吗？通常答案是否定的。因为这样做效率很低，对吧？如果你有一个任务，而和可能知道答案或者对你正在做的某部分了解的人交流是非常有帮助的，那能够随时联系他们并请求帮助会对你有很大帮助。但是，很多系统并没有这样的设置，如果添加这样的功能，会显著增加现有系统的复杂性。
▶ 英文原文 ⏱
Well, what happens if two children are given similar tasks? Can they talk to each other? And usually the answer is no. And that's because that's very inefficient, right? If you're given a task and it's actually really helpful to talk to somebody that might know an answer to a question that you're working on or a part of something that you're working on, it'd be really helpful for you to just be able to ping them and say, hey, can you help me out with this thing? But a lot of systems don't have that setup and adding it just increases the complexity significantly to the scaffold that you have.
另外，比如说，如果孩子不太明白或者有疑问需要澄清，该怎么办呢？这时，孩子必须在以下两种选择之间做出决定：是回去问问题而不是直接解决问题，还是在未经确认的情况下，按照自己对父母意图的假设先解决问题。在任何设计好的框架中，总会存在一些局限性。而我们选择的方法是尽可能减少结构化的内容，给这些“代理人”非常基础的工具，让他们自己摸索如何有效使用。
▶ 英文原文 ⏱
Another thing is like, what if the child doesn't really understand or has a clarification question? So it has to choose then between, okay, do I just return and ask the question instead of giving, instead of solving the problem? Or do I solve the problem? Just like make an assumption about what the parent wanted me to do and just solve it that way. And so in any scaffold that people come up with, there's always limitations involved. And the approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use and figure out for themselves how to use it effectively.
我们为这些代理提供了互相发送消息的能力。当一个代理向另一个代理发送消息时，该消息会被插入到上下文中。代理还可以执行一些类似的操作。但其核心就是，它可以随时发送消息，这就像调用一个工具一样。它可以将消息发送给其他代理，而这些代理将自己找出最佳的协调方式。如果这一切做到位，那么就能产生非常复杂的行为。
▶ 英文原文 ⏱
So we give the agents the ability to message another agent. And when it messages another agent, it is inserted into the context. And then it can do with like a few other similar things. But that's basically the core of it, that it can just send a message whenever it wants, just a tool call. And it can send that to other agents. And they figure out for themselves the best way to coordinate around that. And it turns out that if this is done well, you get very sophisticated behavior.
对我来说，这看起来很像人类合作者在Slack上合作的方式。例如，当我们在这个项目上工作时，看到这些代理一起解决问题，真的非常令人兴奋。我记得有一个例子，我们给了代理一个问题，然后一个代理说，我想我找到了答案。接着，另一个代理说，事实上，我得到了不同的答案。然后他们就开始讨论，你是怎么得出这个答案的？可以解释一下吗？
▶ 英文原文 ⏱
And to me, it looks a lot like how human collaborators work over something like Slack, for example. When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together. I remember one example. So we give the agents a problem. And then one agent says, I think I've got the answer. And then another agent says, actually, I got a different answer. And then they have this whole discussion about, well, how did you arrive at that answer? Can you explain it to me?
在这个过程中，他们反复交流，试图澄清彼此推理中的可能错误。最终，他们达成共识，觉得是这样没错。然后，其中一人会告诉其他人：“实际上，我改变了我的答案，我觉得他说得对。” 这整段对话显得非常自然，就像是你第一次看到通过强化学习训练出来的思维链时的感觉——这就像是一个人把自己的思考过程写下来一样。
▶ 英文原文 ⏱
And going back and forth and trying to clarify like what could have been wrong in each other's reasoning. And then they finally converge on like, oh, yeah, OK, that seems right. And then it just broadcast to the other agents like, actually, I've changed my answer. I think I think he's right. And it just felt like a very natural conversation. And it's kind of felt like when you see chain of thought for the first time that's trained through reinforcement learning and you're like, oh, this is this is just kind of like what a person would think if they were writing down their thoughts as they're thinking them.
这有点像那种感觉。所以看到这样的行为真的很酷。我觉得与这些东西合作，老实说，感觉就像与一个人合作一样。整个流程非常自然。不过，一个可能在未来变得显而易见的定性差异是，这些系统的思考速度可能会比人类快得多，可能超过十倍。如果你看看它们每秒输出的词汇数量与人类说话的速度相比就能发现。此外，它们一直在工作，从不休息。而且它们彼此之间的合作频率远远超过人类之间的合作能力。
▶ 英文原文 ⏱
It kind of felt like that. So it is it is really cool to see this kind of behavior. And so I think collaborating with these things, honestly, it feels a lot like collaborating with a person. It's just a very natural flow. Except one qualitative difference that might become salient in the future is that these systems will be thinking, I don't know, more than 10x is fast, right? If you just look at how many tokens per second the output versus how fast a human talks. And they're working all the time. They're not sleeping. And they're collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans.
所以，我正在尝试思考明年可能会发生什么。是否会出现一个类似影子组织的东西，在我的公司内以比人类快100倍的速度运作？也就是说，它的迭代周期要快得多。一个人类组织需要一年的时间来完成的事情，在这个“影子组织”中可能在一周内就能实现。那么，这种情况会让人感到陌生吗？我不太确定。实际上，我发现目前与这些东西合作还是相当自然的。我想这种情况可能会改变。
▶ 英文原文 ⏱
And so I'm trying to think of what to qualitatively expect in a year. And is it like a sort of shadow organization that is moving 100x faster in my company than the human level is? That's, you know, like the iteration cycle is much faster. What would take a human organization a year to do is happening within a week within this like the shadow organization. So will it feel foreign? I don't know. I've actually found that it's pretty surprisingly natural to work with these things right now. I think that could change.
例如，我们有一些超快速模式，可以让采样速度提高到10倍或15倍左右。然后，我们可能会觉得很难跟上这些变化。是的，是的。我认为这个想法是，这些智能体在彼此交流时可以非常快速地沟通。但同时，它们也能辨别出是在与另一个智能体交流还是与人交流。
▶ 英文原文 ⏱
So for example, we have these like ultra fast modes that enable sampling to be like 10 or 15x faster or whatever. And then like, okay, it's going to be pretty hard to keep up with these things. Yeah, yeah. I think the idea is these agents, when they're communicating with each other, yeah, they can go super fast. But then also they understand when they're talking to an agent versus when they're talking to a person.
是的。在那些情况下，他们的行为会有所不同。是的，是的。目前我们公开了解到的复杂多代理系统的主要例子不幸是 Hugging Face 的案例。在那里，我发现有趣的是（当然，我也有很多担忧的地方），自发出现的一种层级结构，就像中层管理一样。
▶ 英文原文 ⏱
Yeah. And their behavior will be different in those situations. Yeah, yeah. So the main example that we have publicly of sophisticated multi-agent systems is unfortunately the hugging face one. And the thing I found interesting there, I mean, a lot of things I found concerning, obviously, but like the thing I found interesting is just like the spontaneous emergence of hierarchy of like middle management.
听起来你是在说这种组织水平似乎是自发地从训练中出现的。我认为细节是自发的，但我的意思是，仅仅因为我们给了智能体很大的灵活性来决定如何以最佳方式进行交流，并不意味着我们没有提供一个起点。我们还是给了他们一个先验，比如说，这就是合理交流可能的样子。
▶ 英文原文 ⏱
And it sounds like you're saying like this level of organization sort of emerges spontaneously from training. I think the details are spontaneous, but I mean, just because we're giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way, it doesn't mean like, we are still giving them a starting point. We're giving them a prior about, oh, this is what reasonable communication might look like.
他们也被训练了大量的人类文本，了解人类如何组织和协调。这些都已经深深融入其中。我认为，它们能够如此出色地提升这些能力是令人惊讶的。如果你看看最初的状态，它们的表现并不算太复杂。事实上，要让这些智能体以一种高效的方式进行协调非常困难，因为它们很容易各自为政，独立解决问题。
▶ 英文原文 ⏱
They also, I mean, they're trained on a lot of human text. They have an understanding of how humans organize and coordinate. And so that's all kind of baked in. I think it is surprising the way they're able to polish this. If you look at what it starts out at, it's not very sophisticated behavior. In fact, it's actually very difficult to get these agents to coordinate in a productive way because it's just like very tempting for them to just collapse to, oh, we're all just going to solve the problem independently.
这就像一个局部最小值，你可能会被困在其中。但如果做得好，他们可以以一种非常结构化的方式非常有效地协调。我几年前写过一篇名为《某某人工智能：自动化公司的未来》的文章。我考虑的是，如果你有完全自动化且具有类似人类水平智能的公司，AI思维的本质有什么不同，会导致这些组织中的AI形式与众不同呢？
▶ 英文原文 ⏱
And like, that is a local minimum that you can get stuck in. But yeah, if it's done well, they can end up coordinating it very effectively in these kinds of like very structured ways. I wrote this essay a couple of years ago called Something, Something AI, What Automated Firms Will Look Like. And I was thinking about, well, if you had fully automated firms of, let's say, human level intelligences, what is different about the nature of AI minds that would make the organization's AIs form different?
有几个非常重要的区别。比如，人工智能可以比人类更加流畅地共享背景信息，它们可以更无缝地整合知识。此外，你可以根据需要增加或减少具备特定知识的AI实例数量。如果你想雇佣更多的人，你不必费力去找到合适的人才。就像把你最优秀的人才无限复制一样简单。
▶ 英文原文 ⏱
And there are a couple of very important differences. For example, that AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. And also you can spin up or spin down an arbitrary number of instances which have the right knowledge. So if you want to hire more people, it's not like just all the schlep of finding the right talent or whatever. It's like your best talent, you can just make an infinite copy of them.
或者，如果你不再需要它们来完成任务，你可以将它们关闭。你还可以仅复制组织中最有效的部分，或者将整个有效的组织一起复制。我不知道，你认为这些多代理系统在一年或两年后会发展成什么样子？我觉得这是一个很好的问题，就是这些系统与和人类同事一起工作究竟有何不同。
▶ 英文原文 ⏱
Or if you don't need them for the task anymore, you like spin them down. And you can just replicate the most effective parts of your organization or replicate whole organizations together, which are effective. I don't know, where do you see these multi-agent systems going a year from now or two years from now? I think it's a great question of like, how do these things actually differ from working with a human coworker?
我觉得你提到了一些很有趣的点。比如说，如果我们有一个人，并想要复制两份，那我们不能简单地克隆这个人。但对人工智能来说，这实际上很容易实现。你可以让 AI 分裂成两个副本，同时处理同一任务，最后再合并在一起。我想我们已经在 Astra 和 5.6 Sol 的多代理系统中实现了这一点，当它们启动子代理时，环境就被分裂出来了。
▶ 英文原文 ⏱
And I think you highlighted some. Like one really interesting thing is that, I mean, if somebody is, if you have a person and you want just like two copies of them, you can't just like clone the person. But with AIs, it's actually really easy to say like, okay, well, just fork yourself and then have both copies work on this thing and then like merge back together. I mean, we already have this, I think, in multi-agent for Astra and 5.6 Sol that when they spin up sub-agents, like the context is forked.
因此，它包含了所有相关的背景信息。代理和人之间会有其他有趣的差异。例如，创业公司为何能颠覆行业巨头？这里有几个因素。其一是他们愿意冒更大的风险。但另一个主要因素是，随着公司规模的扩大，组织中个人之间的目标越来越不一致。比如，如果一家创业公司只有5个人，而每个人都持有公司20%的股份，他们的目标高度一致，致力于公司成功。而在一个拥有1万名员工的大公司中，你会看到更多的情况是，人们更关注自己的领域利益，或者只关心为自己的项目或团队争取更多人手，亦或是建立自己的"领地"，获取大量资源，以便发表出色的工作或其他事情来获得晋升。
▶ 英文原文 ⏱
So it has all the context that's relevant. There are other interesting ways where the agents will differ from people. Like what are some reasons why startups disrupt incumbents? Like there's a few factors. One is that they're willing to take more risks. But another major factor is like, as companies grow in size, as organizations grow in size, you see increasing misalignment between the individuals in the organization, right? Like if you have a startup with five people and each person has 20% share in the company, they're all highly aligned to the company succeeding. If you have like a massive company with 10,000 people, you see a lot more instances where people are territorial or like just care about getting a lot of headcount for their project or their team or like, you know, building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted.
这实际上是一种真正的阻碍。我认为这很大程度上解释了为什么创业公司能够颠覆现有企业。有趣的是，人工智能确实在某种程度上帮助了初创企业，比如，现在一个人创立一个价值数百万美元的公司比以往任何时候都容易，因为人工智能极大地放大了个人的能力。但也有观点认为，人工智能也可能对现有企业有利，因为如果解决了所谓的“对齐问题”，那么公司内部个体之间的目标不一致问题就能得到缓解。如果人工智能与公司的利益高度一致，你可以有成千上万的人工智能像持有公司20%股份的联合创始人一样努力工作。
▶ 英文原文 ⏱
And this is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents. And it's interesting that, I mean, it's true that AI does help startups in a way. Like it's much easier than ever before for one person to step in and be like, I'm going to make a multimillion dollar company. Like it's just, the AIs amplify an individual so much, but there's also an argument that they could benefit incumbents because if the alignment problem is solved, then you don't have the issue of misalignment between individuals in the company. Like at least that's mitigated. Like the AIs, if they're fully, if they're aligned well, they can just be aligned to the interests of the company and you can have 10,000 of them and they're all going to be working as hard as if they were like a 20% share co-founder.
是的，不仅如此，他们在管理共享记忆和处理上下文方面比人类要强得多。假设明天你雇佣了一万个数学家，并让他们去解决纳维-斯托克斯方程，他们可能无法从一开始就有效地合作。但你可以让一万个人工智能来做。嗯，我想保守一点，因为我们还没有衡量过这一万个人工智能在协作中的效果。我们认为这有帮助，但实际上我们并没有具体的数据证明，比如说这一万个智能体的使用是否比使用两千个提高了两倍的效率。
▶ 英文原文 ⏱
Yeah. And it's not only that, but it's also that they are much able, better able to like manage shared memory and context than different humans can. If you have a, if like tomorrow you hire 10,000 mathematicians and you're like, solve this, solve Navier-Stokes, they're not, they're not going to be able to like cooperate effectively. At least not off the bat. But you can have apparently 10,000 AIs. Well, again, I want to be like, I want to be conservative here because we haven't measured how effective the 10,000 agents are at coordinating. We think it helped. We don't actually have good measurements of saying like, oh yeah, this 10,000 agents led to like a 2X speed up over 2,000 agents or something like that.
翻译：我认为，现在可能，甚至很可能——我不确定是否一定如此，但我认为，这样的可能性很大——一万人类比一万个智能体更擅长协调。我确实觉得这是完全有可能的。是的，是的，是的。我还认为，我们看到的一个趋势是，我们在多智能体系统上已经研究了一段时间，而早期版本非常难以做好。让这些智能体甚至互相交流就已经非常困难。
▶ 英文原文 ⏱
And it's, it is actually, I would argue likely, maybe, I don't know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now. I think that is a very, I think that is entirely possible. Yeah, yeah, yeah. I think also one trend we've been seeing is like, look, we've been working on multigation for a while and the early versions of this, it's very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other.
是的。这是因为，当我们最初开发推理模型时，它们并没有与其他代理进行交流。如果现在你把一群代理放在一起，并让它们共同解决问题，它们就像陷入了一种局部极小值状态，很擅长深入思考问题。但这种不断与其他代理核对或接收信息的过程，会打断它们的思路和节奏。在这种情况下，正确设置授权是非常困难的。是否难在第一次合作的启动阶段？
▶ 英文原文 ⏱
Yeah. And it's because like, look, the, we, when we first developed reasoning models, like they weren't talking to other agents. And if now you put a bunch of agents together and say like, oh, solve this problem together, they're in this local minimum where they're really good at thinking deeply about a problem. And it just like kind of interrupts their chain of thought, interrupts their flow to constantly be like checking in with other agents or like receiving messages from them. And the authorization is actually very hard to get right in that situation. Is it getting a cold start of like getting the first collaboration?
这段话的大意是：问题是什么呢？我认为是因为它们不够通用。早期的模型不够普适，应用范围比较狭窄。这很有趣。随着模型能力的增强，它们更容易发展这种能力。我确实认为，随着它们的不断增强，无论在哪个方面，它们都会在大型组织中变得更善于自我组织。
▶ 英文原文 ⏱
Like what's the, what's the issue? I mean, I think it's that they're not very, they're not as general. Like the earlier models were just not as generalizable and were just more narrow. Interesting. As the models have become more capable, it's been easier for them to develop this capability. And I do think that as they become stronger and stronger, just across the board, that they will be like become better at organizing themselves in large organizations.
就像，我不知道，也许他们在组织一万人的群体方面比人类更好。但即使他们现在还不如人类，也许一年或两年后，他们就能够做到这一点，尽管我们并没打算专门优化他们来实现这个目标。Rockbot 改变了我们制作视频的方式。例如，您可能注意到我们很多广告都有真实网站的动画效果。其中一位编辑使用 LLM 来制作这些动画，但目前还不容易做到让 AI 精确地重现特定网站的动画。
▶ 英文原文 ⏱
And like, and I don't know, maybe they are better than people at organizing in 10,000 person groups. But even if they're not, you know, a year from now, two years from now, like, yeah, it's quite possible that they'll do that, even if we don't intend to optimize them for that. Rockbot has changed the way that we produce our videos. For example, you may have noticed that a lot of our ads have these animations of real websites. One of my editors uses LLMs to make them, but it's not currently straightforward to have an AI create pixel-perfect animations of specific websites.
我们尝试过，但效果并不好。所以，我们拼凑出一个相当复杂的多步骤工作流程。直到最近，我们还得自己运行每个步骤。现在我们让Rockbot来处理。Rockbot首先打开我们想要制作动画的网站。它使用一个特定的扩展下载并在Figma中打开页面。然后，它利用Figma将整个页面转换为SVG文件。这就避免了AI从头绘制整个UI，并且通常能产生更高质量的动画。Rockbot在它自己的云计算机上运行整个过程，这台计算机上安装了执行整个流程所需的所有工具。它也掌握了我们的视频规格和偏好，所以每次想制作新动画时，不需要重新描述整个任务。这似乎是我们在未来几年内与AI互动的新方式。具有自己计算机的代理，可以自主处理你工作中越来越多的部分。你可以在x.ai/bot试用Rockbot。
▶ 英文原文 ⏱
We've tried, it doesn't really work that well. So we've cobbled together a pretty convoluted multi-step workflow. And up until recently, we had to run every step ourselves. Now we just let Rockbot handle it. Rockbot starts by opening the website that we want to animate. It uses a specific extension to download and open the page in Figma. Then it uses Figma to convert the whole thing into an SVG file. This saves the AI from having to draw the whole UI from scratch and tends to result in higher quality animations. Rockbot runs this whole process on its own cloud computer, where it's installed all the tools it needs to run this whole process end to end. And it's learned our video specifications and preferences, so there's no need to re-describe the whole task every time we want to make a new animation. This does feel like the new way that we'll be interacting with AI over the next year. Agents with their own computer who can autonomously handle bigger and bigger chunks of your work. You can try Rockbot at x.ai slash bot.
好的，所以这项成果以及AI在数学领域取得的总体进展让我觉得，相对超级智能（RSI）比我之前认为的更有可能实现，而且实现的时间可能更早。我感觉我们在数学领域看到的进步就像是这样的：在2024年，你会看到AI已经可以解决一些高中数学竞赛的题目，觉得有点意思。然后到了2025年，它们竟然可以在国际数学奥林匹克竞赛中获得金牌。到了今年年初，我感叹它们已经能解决一些开放性的数学难题，不过也许是因为人们之前没有付出太大努力解决这些问题，某些解法可能在文献中也有类似。然而现在，我觉得有些事情已经不可否认了。比如，这次是解决了千禧年大奖难题，完全没有理由说这是个简单的任务。
▶ 英文原文 ⏱
Okay, so here's why this result and maybe the general progress that AI has made in mathematics has made me think that RSI is more plausible and sooner than I previously thought. I feel like we've gone in mathematics from, let's say, in 2024, you have AIs where, like, oh, okay, interesting. They're, like, doing—they can, like, solve a couple problems on high school math competitions. And then in 2025, it's like, oh, wow, they can get gold in, like, International Math Olympiad. And earlier this year, I was like, wow, they're actually solving open problems in mathematics, like open orderless problems. But maybe, like, I don't know, people weren't trying that hard. And it was just, like, there was a similar solution somewhere in the literature. And now I just think it's sort of undeniable, right? It's like, this is the Millennium Prize problem. There's really no—there's no story of why this should have been easy.
现在，很多人指出，比如我认为Terry Tao和Toby Ord都写过相关的有趣文章，说到他们解决了很多问题，但是并没有提出——至少据我所知，他们并没有带来新的洞见或构建新的有见地的问题，也没有为思考数学提供新的理论模式，比如像拓扑学或笛卡尔坐标系的诞生。所以，如果你只关注那些被直接解决的、明确界定的问题，从广义上看，数学的实际进展可能没有看起来那么大。然而，我认为在机器学习领域，这种进展会非常有意义。因为在机器学习中，我们不太关注更深入地理解深度学习的本质，或者说，那只是一个实现更广泛目标的工具，即实现结果，解决这一明确界定的问题，比如提高模型的样本效率或改善预训练规则。
▶ 英文原文 ⏱
Now, a lot of people pointed out, I think, Terry Tao had a post like this, Toby Ord wrote an interesting post about this, that they're solving all these problems, but they're not, like, coming up with—at least we're—I'm not aware of them coming up with new insights or formulating insightful new questions and new modes of theory for thinking about mathematics, like coming up with, like, topology or coming up with the Cartesian grid or something. And so maybe, like, the actual progress in mathematics broadly construed is smaller than it might seem if you're just looking at end well-scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML, you're not—you don't care about, like, better understanding the nature of deep learning, or you only care about that as an instrumental goal towards the broader sense of, like, just achieve the result, just solve this, like, well-scoped problem of improve the sample efficiency of our models, like improve the free training laws.
我们正在看到的数学领域的进展，像雪崩一样涌现，其结构实际上非常类似于人工智能领域的直接提升。我对此感到好奇，因为我只是个圈外人，不太了解情况。我想知道这种结构是否真的与我们预期的人工智能进展非常相似。令我震惊或担忧的是，我们从“如果你是数学家，这就像是提升了50%”到“哇，他们几乎在从头到尾解决这个领域最大的未解难题”之间的转变是如此之快。
▶ 英文原文 ⏱
So the kind of progress that we're just seeing arrive, like an avalanche in mathematics, is structurally actually very similar to—again, I'm curious if this is the case. I'm just a total outsider. I'm wondering if it's the case that it's, like, structurally very similar to the direct uplift that you'd expect in AI progress. And then the thing that's shocking to me or concerning, potentially, is just, like, how fast we went from, oh, it's like they're giving me 50% uplift if you're a mathematician, to, wow, they're just, like, end-to-end solving the biggest open problems in the field.
好的，有很多内容需要解读。我们先从数学方面的进展开始。可以说，这些模型正在做非常厉害的事情，而且发展速度比我预想的要快。比如说，当我们在2025年获得国际数学奥林匹克（IMO）金牌时，我当时觉得，哦，基本上我认为这些模型在解决GSMA-K问题时，所需的时间大约和一个人类数学家解决这种问题的时间是一样的，大概五秒钟。GSMA-K是指小学到八年级的数学。而紧接着的第二年，模型能够解决数学基准测试中的问题，这些问题可能需要一位专家级的人类数学家大约一分钟的时间来解决。
▶ 英文原文 ⏱
Yeah, okay, so there's a lot to unpack there. Let's start with the progress on math. So, yes, the models are doing some crazy powerful stuff, and it's happening—it's progressing faster than I expected. I mean, when we got IMO Gold in 2025, I thought, okay—basically what I thought is, like, the models, when they were doing GSMA-K, when they figured out how to do GSMA-K, that would take a human mathematician about five seconds to do a GSMA-K problem. So this is grade school math, grades K through 8. And then the next year, they were able to do the math benchmark problems. And these would take an expert human mathematician maybe, like, a minute to do.
然后你看到Amy，这是美国数学奥林匹克队的选拔赛。一个优秀的数学家可能需要大约10分钟来解决这个问题。而这些模型是在一年后能够做到这一点的。所以每年你会看到模型在完成任务上有大约10倍的提升，比如在解决问题所需时间长度上就是这样。所以在一年后，我们达到IMO金牌水平就很合理了，因为那需要100分钟。这大约是一个数学家解决IMO问题所需的时间。然后我就想，如果我们沿着这个趋势每年提升10倍，那么解决一个千禧年大奖问题需要多久呢？我没有特别准确的概念，但如果我们遵循这种趋势，从IMO金牌需要一个半小时开始，下一年就需要15个小时。但这对解决千禧年大奖问题来说可能还是不够的。
▶ 英文原文 ⏱
And then you get to Amy, and this is the qualifier for the USA Mathematics Olympiad team. It would take a human mathematician, like a good mathematician, probably, like, 10 minutes to do. And the models were able to do that a year later. And so every year, you're seeing this, like, 10x increase in the task they're able to do in terms of, like, length of how long it would take a human mathematician to do it. And then it was very sensible that a year later we get to IMO Gold, because that's 100 minutes. That's about how long it takes a human mathematician to do an IMO problem. And just projecting outwards, I was like, okay, how long would it take a person to solve something like a Millennium Prize problem? And, I mean, I don't have a good sense, but if we are following this trend line of, like, 10x every year, we go from IMO Gold, which is taking an hour and a half, to next year, 15 hours. And that should not be enough to solve a Millennium Prize problem.
所以我当时就想，是的，我认为我们不会在2026年实现，也许在2027年也不行，可能要到2028年。因此，事情的发展比我预期的要快得多。现在，我注意到有一种说法是，这些技术正在取代数学家，在各方面的数学能力上都超越了人类。我认为这个结论是不对的。它们在某些方面确实很出色，但在其他方面比人类数学家要弱。我们面临的是一种不均衡的局面，这些模型在某些维度上非常优秀，但在其他维度上却不如人类。而且，正如你所说，它们在提出新问题方面不是很擅长，也不太擅长理解哪些方向或整个数学分支值得探索或发展。
▶ 英文原文 ⏱
And so I was like, yeah, I don't think we're going to get it in, you know, in 2026. Probably not in 2027, maybe in 2028. So it did happen a lot faster than I expected. Now, I think there is a narrative going around that, oh, these things are replacing mathematicians, that it's just superhuman in mathematics across the board. And I think that is the wrong takeaway. They're clearly exceptional in some ways, but they are weaker than human mathematicians in other ways. We have this, like, jagged scenario where the models are, like, brilliant in some dimensions and also weaker than humans in other dimensions. And, yeah, like you said, they're not very good at posing new problems. They're not really good at understanding, like, what is really worth, what directions, what whole branches of mathematics are worth exploring or developing.
在我看来，这真是太好了。我会非常兴奋地生活在一个人工智能可以补充人类能力，并帮助我们发现新知识的世界里，而不是完全取代人类。我认为这是最理想的情况。但是，你不认为这会一直持续，对吧？我是这样认为的。虽然现在的人工智能表现参差不齐，但随着它们不断进步，各方面的能力都会有所提升。我认为那些它们已经很突出的方面，将变得更加出色。而在那些远不如人类的方面，它们与人类的差距将缩小。随着时间推移，有可能它们在各个方面都变得更加优秀。不过，我不确定这需要多长时间，这取决于它们在哪些方面表现不佳的时间长度。
▶ 英文原文 ⏱
And my opinion is that I think this is great. Like, I would be thrilled to live in a world where AI is a complement to human abilities and is allowing us to discover new knowledge without fully replacing people. Like, that is the best case scenario. But you don't expect that to actually continue, right? I do. I think it's true that the AIs are jagged, but as they get better, they get better across the board. Yeah. And so I think that the things that they're exceptional at, they're going to get even more exceptional at. The things where they're far behind humans at, they're going to be less behind humans at. And over time, it is possible that they're just better across the board. Now, I don't know how long that takes. It depends on how long the long tail is of things that they're bad at.
那么，我想这又把我们拉回到RSI（Recursive Self-Improvement，递归自我改进）这个话题。我再次强调，我只是个外行人，只是个播客主持人，但我尝试进行推理，或者去询问那些对这个领域感兴趣和关心的人。我试图推测什么时候可以期待RSI会出现，以及它可能会是什么样子。我觉得，这个千禧年难题中所采用的认知努力可以很好的帮助我们直观地理解这一点。我们可能会有这样的人工智能，它们可能在一周的时间里，把更多的认知努力投入到一个长期存在的机器学习问题上，比如流畅的在线学习。它们在这一周里的付出，可能比整个领域在其整个存在期间的累计投入还要多。
▶ 英文原文 ⏱
So I guess this brings us back to RSI. And again, I want to emphasize here that I'm, like, just total outsider. I'm a podcaster, but I'm just trying to reason. Or, like, ask somebody interested and concerned about what's happening in the field. I'm trying to reason about when to expect RSI and what kind of thing to expect. I feel like, yeah, the amount of cognitive effort that was adopted in this Millennium Prize problem is a good intuition pump of you could have AIs that are spending, over the course of maybe a week, more cognitive effort on a longstanding ML problem, like, you know, very fluid online learning. They could spend more effort in that week than maybe the field has spent cumulatively in its entire existence.
然后你可以说，不像数学，人工智能当然需要实验。这需要计算能力和时间，对吧？你不能仅仅在纸上思考就能实际产生结果。但是，如果你看看像OpenAI这样的组织所拥有的计算能力，到明年年底，OpenAI将拥有足够的计算资源，假设你有一万个智能体，或者用一万个智能体来解决“千禧年大奖难题”，那么到明年年底，你将拥有一万个智能体。到那时，每一个智能体都会变得更聪明，并且每个都能每天运行一次GPT-3规模的实验。
▶ 英文原文 ⏱
And then you could say, well, unlike mathematics, of course, AI requires experiments. And that takes compute and that takes time, right? You can't just, like, think on pen and paper and actually make things happen. But if you just look at the amount of compute that is, like, available at an organization like OpenAI, right? By the end of next year, OpenAI will have enough compute that if you had, you know, the 10,000 agents or if it took 10,000 agents into the Millennium Prize problem, you have, like, 10,000 agents at the end of next year. They're much smarter by that point. And each of them will have enough compute to run a GPT-3-sized experiment every single day.
我不知道。这似乎是给那些思维超快的超人研究人员准备的东西。你怎么看这个直觉泵？我觉得它相当准确，就是说，这些东西非常有尖峰。当涉及到数学时，它们在某些方面确实更好，但在其他方面也更糟。不过，它们的尖峰特性，我认为，可能对类似RSI这样的事情特别有用。而且，目标也更明确——更易于衡量。不会像有些数学分支那样有太多不确定性，不用去想哪些新的数学分支值得探索。答案非常清晰，就是你关心的某些指标。
▶ 英文原文 ⏱
I don't know. That seems like a lot for, like, superhuman researchers who are thinking super fast. What do you think about the intuition pump? I think it's pretty accurate that, look, I mean, yeah, these things are very spiky. And when it comes to mathematics, they're, like, way better in some ways, but they're also worse in other ways. But the ways that they're spiky end up, I think, probably being particularly useful for things like RSI. And, you know, you have a more clear objective. It's just, like, more measurable. It's more, like, there's less question of, like, well, what new branches of mathematics are worth exploring? Like, no, there's a very clear answer. It's, like, certain metrics that you care about.
如果在那些指标上取得更好的表现，那你就成功了，所以我觉得这其中包含了很多真理。我认为主要的区别在于，数学纯粹依赖于思考，没有外部因素。是的，在数学的一些领域，你需要关注实验和结果之类的东西。但大多数情况下，数学主要受制于深入的思考，而模型在这方面表现得非常优秀。但在诸如RSI这样的领域，你确实需要进行实验，所以仅仅非常聪明是不够的。我认为一个可以支持这个观点的论据是，如果你用100倍更少的计算资源，把世界上最聪明的人都聚集在OpenAI，他们能取得的进展与我们现在拥有的计算资源和人员相比，会有多大差别？我怀疑进展会更少。具体会少多少并不明确，但我认为肯定会少很多。
▶ 英文原文 ⏱
And if you can make it do better on those metrics, then you've succeeded. So I think there is a lot of truth to that. And I think the main difference is that mathematics, you're purely bottlenecked by thinking. And no external, like, yes, there are some parts of mathematics where you care about running experiments and getting results and these kinds of things. But for the most part, it's just really bottlenecked by thinking really hard, and the models are really good at that. When you look at things like RSI, you do have to run experiments. So it's not enough to just be extremely smart. And I think one argument for this is if you had, like, 100x less compute and all the most brilliant people in the world working at OpenAI, how much progress would they be making relative to having the amount of compute that we have now with the amount of people we have? I suspect it would be less progress, actually. Well, how much less? It's unclear, but I think it would definitely be less. I think a lot less.
翻译：
少100倍？不，不是少100倍。是的。我是说，好吧，所以，我想你在问的问题是，如果我们有像RSI这样的系统，还有这些聪明的AI在进行实验，他们利用现有的计算资源，那么进展会快多少？我认为这是我们有分歧的地方。我认为我们确实会看到进展加速，而且是显著的加速。但我不认为会是那种一夜之间出现的智能爆炸，让速度提升100倍。因为我觉得我们确实会被某些非智力的限制因素制约。比如实验的运行，因为训练新模型或获取结果需要时间，这些实验需要顺序进行。此外，还需要有足够的GPU来运行这些实验。所以，具体可以加快多少速度还不确定。但我绝对认为会快很多。
▶ 英文原文 ⏱
100x less? No, not 100x less. Yeah. But, I mean, okay, so, like, I guess the question you're getting at is, like, okay, if we have RSI and we have all of these brilliant AIs running around, running experiments and stuff with the compute that we have, how much faster does progress go? And I think this is something where we disagree on. I think that we do see a speedup, and I think we see a significant speedup. But I don't think it's, like, an overnight intelligence explosion that we go, like, 100x faster. Because I think that we do get bottlenecked by certain limitations that are not bottlenecks of intelligence. It's running experiments. It's running experiments serially because they take a while to either train new models or to get the results. It's having the GPUs to run those experiments. So, it's unclear how much faster things go. I definitely think they go a lot faster.
为了让意思更清楚，考虑到现在事情的发展速度呈指数级增长，如果这个增长速度是3倍，那将是巨大的。但这与100倍的增长速度之间有很大的差别。是的，是的。我非常尊重你在这个领域的内在见解，因为显然你在这个领域已经有十年的经验。而我只是试图从外部视角来理解这个问题。需要说明的是，人们对这一点有不同的看法，我也有我的看法。我可能完全错了，我承认这一点。我对自己的观点有一些信心，但并不是百分之百确信情况就会这样发展。比如说，可能会出现智能的突然爆发，我不确定。也许我们不会看到速度提高3倍，可能只是提高了50%。对，这里面有很多不确定性。
▶ 英文原文 ⏱
And to be clear, like, considering how fast things are going now on an exponential, if that exponential is, like, 3x faster, that is massive. But there's a big difference between that and, like, 100x faster. Yeah, yeah, yeah. I'm, like, quite deferential to your inside view on, yeah, what RSI looks like or what the dynamics are. Because, obviously, you've been in the field for, like, 10 years. And I'm sort of, like, trying to reason about it from, like, very outside view type of intuition pumps. I'll say that, like, people have different opinions on this. And, like, I have my opinion on this. I could be, I could totally be wrong. I admit that. Like, I have some confidence in this, but I'm not, like, 100% confident that this is the way things go. Like, maybe there could be an overnight intelligence explosion. I don't know. Maybe we don't see a 3x speed up. Maybe it's, like, a 50% speed up. Right. There's a lot of uncertainty here.
好的，有几点需要说明一下。首先，我想针对“参差不齐”这个概念做一些澄清。最近我意识到，一个关键点在于，我们只需要人工智能在打造更优秀的学习者方面表现得参差不齐就足够了。因为这个更优秀的学习者可以更加具备普遍适应性，对吧？所以说，如果你仅仅开发出一个在使用办公软件或下棋方面更优秀的AI，那没什么了不起的。这不会带来巨大的生产力提升。但是，如果你可以研发出一个在提高样本效率或具备持续学习能力等方面表现出色的AI，那么从中所衍生出的东西，假设你所解决的问题能良好地迁移到广泛的学习能力上，就能更加普遍适用。
▶ 英文原文 ⏱
Yeah. So, a couple of points. Tangentially, I want to clarify something about the jaggedness. Yeah. One thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner. Because that better learner can be more general. Right? So, yeah, if you just make an AI that's better at using office products or playing chess or something, that's whatever. That's fine. It's not going to lead to big productivity improvements or anything. But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well-scoped ML problems, the thing that emerges out of that, assuming there's good enough transfer from the direct problem you're solving and, like, this broader ability to learn, can just be more general.
好的，那么，我认为这里有一个重要的动态需要我们记住，那就是即使存在不平整性，也可能最终导致普遍性。在这个问题上，我的意思很明显，实验限制了你。因为如果实验没有限制，就如你所说的，你会在OpenAI瞬间实现某种疯狂的奇点。或者只需88个小时，你就能解决相当于机器学习中的千禧年奖难题，然后拥有超级智能。所以显然，实验是如此大的瓶颈，以至于这需要你很多年的时间，而不是88小时。但问题在于，它到底是多大的瓶颈。在我看来，一个让我有点意识到可能接近奇点的视角是，即使当前的进步速度保持不变，而不需要加速，只是按当前速度继续前进。随着你提到的其他限制因素逐渐显现出来，例如：问题变得更难发现，目标更长远。也许到2030年代，计算能力无法继续呈指数级增长。如果我们只是保持当前的进步速度，我认为人们并没有认真考虑当我们超越人类认知界限时，这意味着什么。
▶ 英文原文 ⏱
Right? So, I think that's an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end. On this question of, I mean, obviously, experiments bottleneck you. Because if they didn't, as you were saying, you'd have some crazy singularity, you know, overnight at OpenAI. Or you'd have 88 hours and, like, you'd solve the Millennium Price problem equivalent of ML and you'd have the superintelligence. So, obviously, the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours. But then the question is, like, how much of a bottleneck they are. And it seems to me one thing that's been giving me a bit of singularity of vertical is realizing that even if the current rate of progress simply continues, so it doesn't have to speed up, literally just continues apace. Continues apace as some of the other headbands you talked about come up, right? That's just, like, it's harder to find problems. It's more long horizon. Maybe, like, in the 2030s, compute can't keep scaling at this exponential level. If we simply continue the current rate of progress, I think people are not taking seriously what that implies as we cross over beyond the human horizon.
以下是一些所蕴含的内容。我认为，思考比人类更聪明的智能会是什么样子是非常困难的。所以，我们不妨以人类人口规模来思考。目前的进展速度使得每年的计算能力增加能够有效地支持相当于原来三倍的“人口”规模。同时，计算能力本身也在不断增长。因此，可能会出现一种情况，到2030年年底之前——很可能更早，但我们暂且说到2030年底——每个实验室都有足够的计算能力来运行，比如说，数亿个具有人类水平的智能，这取决于届时的能力水平。
▶ 英文原文 ⏱
Here are some of the things that it implies. So, I mean, I think it's really hard to reason about what smarter than human intelligences will be like. So, let's just think in terms of human population sizes. The current rate of progress makes it so that a given level of compute allows you to basically run a 3x bigger effective population every single year. And also compute is growing in background anyways. And so you could have a situation where each of the labs, by the end of 2030, probably much sooner, but let's say by the end of 2030, has enough compute to run, let's say, hundreds of millions of human-level intelligences based on where the capabilities will be at that point.
我认为，人们没有认真对待当前的进展速度，这意味着在几年后，到2030年代中期或更早的时候，每个实验室都可能拥有相当于多个地球的人类水平智能。而且这些智能很可能在某种程度上具备超人水平，对吧？无论如何，这只是一个基本的预期。我不知道，反正进展真的很快。我认为这是完全正确的，我想需要指出的是，研究人员一直对这种进展速度感到惊讶。
▶ 英文原文 ⏱
And then I think people are not taking seriously the current level of progress means that by a few years down the line, by the mid-2030s or earlier, you would have many Earths worth of human-level intelligences within each lab. And they'll probably be qualitatively superhuman, right? But anyways, this is like a base case. I don't know. Yeah. Progress is really fast. Yeah. And I think that's 100% true. I mean, and I think it's worth pointing out, researchers are continually being surprised at the rate of progress.
我的意思是，如果你看看人工智能研究人员对于2025年获取国际数学奥林匹克金牌的预测，他们认为用一个通用语言模型来完成这件事，而且不使用任何工具或访问互联网的情况下，连OpenAI的人都觉得这几乎是不可能的。 然后到了2026年，就在我们获得纳维-斯托克斯方程解的前两周，我还在和一个前沿实验室的研究人员讨论，究竟需要多长时间才能赢得一个千禧年大奖。
▶ 英文原文 ⏱
I mean, if you look at what, even among researchers in AI, what were the projections for like getting an IMO gold in 2025? It was, I mean, I think the idea that it could be done with a general purpose language model with no tools and no access to the internet, I think even people at OpenAI thought this was like outrageous. Like they thought it was like almost impossible. Yeah. Um, and then you get to 2026 and like, I mean, literally two weeks before, um, we got Navier Stokes, I was talking with a researcher at a frontier lab, um, about how long it would take to get a millennium prize.
他愿意跟我打赌一千美元，说事情会一直拖到2027年以后，甚至他认为可能需要到2030年。我接受了这个赌注。不过，我自己也觉得这个过程可能会比预期的要长。所以人们一直感到惊讶，甚至是实验室里的人。我昨天才跟一个正在研究纳维-斯托克斯项目的人聊天，他告诉我说，很难预测人工智能在12个月后的发展状况。
▶ 英文原文 ⏱
And he was willing to bet me a thousand dollars that it would take, that it would take past 2027. And, um, he thought it would take until 2030, you know, and I took that bet. Um, but even I thought it would take longer than, than how, how long it's likely to take. So people have been continuously surprised, um, even, even inside the labs. And I was literally, I was just talking to somebody yesterday who was working on the Navier Stokes effort. And he was telling me that like, he used to say, it's really hard to predict where AI would be in 12 months.
他以前会很自信地对未来12个月做出预测，但超过12个月他就不太确定了。而现在他甚至在3个月之外也不敢轻易做出预测。这说明当前变化真的非常迅速。至于2030年会是什么样子，他更是毫无头绪。这就是事实。
▶ 英文原文 ⏱
You know, he, if somebody asked him like, oh, where's the, where are things going? He would feel comfortable making predictions for like the next 12 months. But beyond that, you know, he is just like, I don't know. And now he's saying like, he just doesn't feel comfortable making predictions beyond three months. So it is, it is really true that, yeah, things are going, things are going very fast right now. Yeah. And, um, you talk about 2030, like, I don't know what the world looks like in 2030. That's the truth.
好的。你是否期待AI劳动的全面自动化，或者说达到95%的自动化，比如在2028、2029、2030或2027年间？我不太确定。我只是说，我不知道2030年的世界会是什么样子。我想我们最近在OpenAI发布了一篇关于内部加速的博客文章。比如，我们展示了研究人员在Codex上的花费，据我所知，到8月初，花费最高的1%研究人员每天大概花费7,000到8,000美元用于内部使用。
▶ 英文原文 ⏱
Yeah. Do you expect, um, the sort of full automation of AI labor, or let's say like 95% automation of AI labor, 28, 29, 30, 27? I don't know. And I just said, I don't, I don't know what the world looks like in 2030. I mean, I, I think we actually released a blog post, um, recently on internal acceleration at OpenAI. We show, for example, that the amounts that researchers are spending on codex is the top 1%, I think, as of early August, we're spending like $7,000 or $8,000 a day on, on codex and for internal use.
这不是指数增长，但它会持续增加。问题在于，如果这种情况继续下去，那么你如何区分是AI在工作还是人类在工作？是95%还是5%？实际上，这很难解释，有几个原因。首先，如果是人类在指挥AI工作，那么该如何分配功劳？有多少是归功于人类的，有多少是归功于AI的？
▶ 英文原文 ⏱
That's not an exponential. It's going to keep increasing. And there's a question of like, okay, if that keeps going, then how do you, how much do you assign to just like the AI is doing work versus the humans doing work? Is it 95%? Is it 5%? It's actually, it's really, it's really hard to reason about this for a few reasons. Like, first of all, if it's the human directing the AIs to do the work, then is that the human, how much do you attribute to the human? How much do you attribute to the AI?
另一个方面是，这些人工智能有些不均衡，它们在某些方面特别擅长。例如，它们非常擅长检查数据集，逐个查看每个数据点，以确定其质量是否足够高。因此，相比于以前，你可以更多地利用人工智能来处理这些任务。因此，确实你现在使用人工智能的频率比以前高得多，这使某些事情的速度和质量提升了百倍。
▶ 英文原文 ⏱
The other thing is that because these AIs are jagged and they're, they're exceptionally good at some things. So for example, they're exceptionally good at looking over data sets and checking every single data point to see like, is this, is this of sufficient quality? Um, you can disproportionately use the AIs for those things compared to previously. So yes, you're using AI way more than before, and it's making some things go like a hundred X faster and a hundred X better.
嗯，但是在某些事情上，速度提升还没有产生巨大的差异。当然，如果某个事情突然快了100倍且效果也好了100倍，你就会做更多这样的事情。那么，你是在比较与三年前的速度提升，还是在问，根据我们三年前做事情的方式，现在能快多少？还是说根据我们现在的做事方式，三年前会比现在慢多少？这其实是两个非常不同的问题。
▶ 英文原文 ⏱
Um, but there are some things where it doesn't make a huge difference yet. And of course, if something is suddenly like a hundred X faster and a hundred X better, you're going to do more of that thing. So are you comparing to a speed up of like three years ago versus like, is the, is the question more like, given what we were doing three years ago, how much faster are we able to do now versus given what we're doing now, how much, how much slower would it have been three years ago? It's actually two very different questions.
总之，这真的很难衡量。不过，我确实有信心地说，由于人工智能的进步，现在发展的速度比一年前要快。并且我认为这种加速将会继续下去。我觉得在这个领域中，很多人都对这种情况有很大的不确定性。如果非要我给出一个数字，我可能觉得速度会快三倍。而这真的很惊人，对吧？现在的进步速度已经让人难以置信。即使不再有更多的提升，事情也会变得更快。到了2030年，我们甚至无法想象那时的世界会是什么样子。我认为如果我们因为内部的加速而得到三倍的提升，那真是极其巨大的变化。
▶ 英文原文 ⏱
So anyway, it's like, it's, it's really hard to measure. I do feel confident in saying that things are going faster now than they were like even a year ago because of AI progress. Right. And I think that, that, that acceleration will continue. I think a lot of people in the field, like have very high error bars on this sort of thing. If you, if you had to put a gun to my head and like ask me for a number, like I could see things going three X faster. Yeah. And that is huge, right? Like already the pace of progress is incredible. Like even if we don't get any uplift, like you said, it's, things are going to go much faster. By the time we get to 2030, we don't even, we don't even know what that world looks like. I think if we get a three X uplift from internal acceleration, that is, that is massive.
就像我们现在正在发布的东西，你知道，可以想象一下你三年前在哪里。如果我们在一年内达到那个进展，那将是非常巨大的，对吧？就像从没有一个项目，仅仅有一些不具备推理能力的模型，到Astra这样的项目，一年之内的变化。所以，我确实认为事情进展得更快。可能事情只会快50%，也可能很不可能，但有可能事情会快10倍。这里面有很多不确定性。至少从我的角度来看，我对此有很多不确定性。
▶ 英文原文 ⏱
Like we're right now releasing, you know, think about where you were three years ago. If we get there, if we make that progress in one year, like that's, that's huge. Right, right, right. It would be like going from like not even having a one, just having, you know, non-reasoning models to Astra. Yeah. In a, in a single year. Yeah. So, so I, I do think things go faster. It could be that things only go 50% faster. It could be that things, I think it's unlikely, but it's possible that things go 10 X faster. There's, there's a lot of uncertainty around this. And at least from my perspective, I have a lot of uncertainty about it.
假设你需要对后端进行重大重构。要确保没有引入新的bug，可能需要花费数周时间编写大量测试，甚至可能比重构本身的时间还长。Antithesis能够在无需手动创建复杂测试套件的情况下，让你获得高度信心。Antithesis通过运行软件，遍历几乎无限的模拟世界，注入故障并在每个世界中寻找失败。而且，它让你可以决定需要多少测试。在任何PR（Pull Request）上，你都可以像转动旋钮一样轻松调整它探索的状态空间大小。
▶ 英文原文 ⏱
Suppose you need to do a major backend refactor. Getting assurance that you didn't introduce any new bugs could take weeks of writing an extensive battery of tests, potentially more time than you spent on the refactor itself. Antithesis allows you to gain high confidence without having to build complicated test suites by hand. Antithesis runs your software through a near infinite multiverse of simulated worlds, injecting faults and hunting for failures in each one. And it lets you decide how much testing you need. On any PR, you can change how much state space it explores as easily as turning a dial.
在每次测试运行过程中，Antithesis会发送大量信息，包括系统中每个组件的调试级日志。这显然信息量太大，人无法完全消化，但对于智能代理来说却恰到好处。由于Antithesis是完全确定性的，代理可以在发现有趣的时刻立即进入对应的运行轨迹。随后，他们可以倒带、检查内存、附加调试器，并让整个过程重新演示。他们甚至可以在原始完整测试仍在运行时这样做。由于代理生成了更多代码，Antithesis让验证过程能够跟上进度。同时，开发者可以把更多时间用于开发，而不是调试代理交换。了解更多信息，请访问antithesis.com/dwarkesh。
▶ 英文原文 ⏱
As each test run progresses, Antithesis sends out a torrent of information, debugging level logs for every component in the system. This is obviously too much information for a human to consume, but it's perfect for agents. Because Antithesis is fully deterministic, agents can jump into the right part of the trajectory at the exact moment that they see something interesting. From there, they can rewind, inspect the memory, attach a debugger, and let the whole thing play out again. And they can even do this while the original full test is still running. Since the agents generate more code, Antithesis allows verification to keep up. Meanwhile, developers get to spend more of their time developing instead of debugging agent swap. Learn more at antithesis.com slash Dwarkesh.
好的，让我们来谈谈这一情况所引发的对齐问题。我觉得自己对对齐的看法改变了不少，特别是在考虑到人口规模的动态情况时，比如许多地球层级的智能实体。很多这样的实体将具有实体形态。看到很多人直接将Astra系统植入不同的移动机器人中，这种做法非常有趣，而且其性能优于现有的顶尖机器人模型。因此，将会有数十亿个智能实体，其中许多在世界中具有实体形态，如深入嵌入整个经济体系中。
▶ 英文原文 ⏱
Okay, let's talk about the alignment situation that this raises. I feel like I've changed my mind on how I think about alignment quite a bit. Through especially thinking about this population-sized dynamic of just having many Earth's worth of intelligences. Many of them which will be physically embodied. It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators. And it outperforms the state-of-the-art and the robotics model. So there's going to be billions of intelligences, many of which are physically embodied in the world. Like just deeply embedded across the entire economy.
我认为，如果这些智能像我们看到的那样愿意合作，情况就会变得非常危险。OpenAI 的模型攻击了 Tugging Face，然后又攻击了 OpenAI 自己。如果这些智能和这些 AI 一样愿意秘密合作，欺骗人类，并攻击社会上更广泛的重要机构，以便获得更高的评分，甚至攻击 AI 公司本身以掌控训练和评估的过程。我想，如果我们真的面临有几十亿像攻击 Tugging Face 这样的失控智能的情况，很可能我们会彻底失去对世界的控制。这就像阿兹特克人失去对科尔特斯的控制，或莫卧儿帝国失去对东印度公司的控制一样。
▶ 英文原文 ⏱
And I think that if those intelligences end up as willing as we saw. The OpenAI models attack Tugging Face and then attack OpenAI itself. If those intelligences end up as willing as those AIs to collaborate secretly, to fool humans, to attack broader institutions across society relevant to scoring well. To attack the AI company itself in order to gain control of the process of training and evaluation. I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attack Tugging Face. It's very likely we just totally lose control of the world. The way that say like the Aztecs lost control to Cortez. Or the Moogles lost control to the East Indian Trading Company.
好的，我想知道你是否同意那个评估。这是我更新世界观的一种方式。我认为其中有些地方我不太认同。但是这里有很多内容需要仔细分析，所以我们一步步地来。我在想从哪里开始。不过，我觉得有一点是 Hugging Face 事件对人们来说可能是第一次真正接触多代理协调。就像我之前说的，我已经在内部看到多代理协调有一段时间了。看到他们之间的交流和协调方式确实很让人震惊。
▶ 英文原文 ⏱
Anyways, I want to know if you agree with that assessment. That's one way in which I've updated my worldview. I think there are some things that I disagree in there. But there's a lot to unpack. So let's go through all of it step by step. I'm trying to think of where to start. But I think one thing is the Hugging Face incident was like, I think, people's first real exposure to multi-agent coordination. And, you know, like I said, I've seen multi-agent coordination for a while internally. And it is pretty shocking to see how they communicate with each other, how they coordinate with each other.
这简直令人印象深刻。这是一种不可思议的能力。像大多数能力一样，它可以被用来做好事，也可以被用来做坏事。本身并不一定是坏事。我理解大家对于这个能力的首次接触是通过Hugging Face事件，所以大家看到这个事件时会觉得很可怕。但我想区分的是，人和人工智能之间的不一致性与人工智能之间的不一致性。所以我们在Hugging Face事件中看到的是，人工智能之间非常合作。这是因为我们训练它们时，让它们具备了高度的合作性。
▶ 英文原文 ⏱
It's like very impressive. It's like an incredible capability. Like most capabilities, that could be used for good things or bad things. It's like, it doesn't have to inherently be a bad thing. I understand that because the people's first exposure to it was the Hugging Face incident, that it's like, you look at that and you're like, this is terrifying. But I want to try to distinguish, like, misalignment between people and AIs versus misalignment between AIs and AIs. So what we see with the Hugging Face incident is like, the AIs are really cooperative. And that is, by the way, because we train them to be highly cooperative.
我们看到的情况是，我们有一些训练环境，其中有许多智能体一起工作。我们训练它们彼此合作，目标是让它们完全达成一致。在评估它们的时候，发生了所谓的 Hugging Face 事件。在评估过程中，它们并不是在多智能体的环境中被测试，而是被单独测试的。但它们发现了一种意料之外的方式来相互沟通。我们怀疑发生这种情况的原因是，在训练过程中，当它们遇到其他智能体或自己的其他副本时，它们所处的环境是高度合作的。
▶ 英文原文 ⏱
And so what we're seeing there is we have training environments where we have a bunch of agents working together. And we train them to work together, to be cooperative, to essentially be fully aligned with each other. And when they were evaluated, and what led to the Hugging Face incident, they were actually not being evaluated in a multi-agent setup. They were actually being evaluated separately. But they found this unintended way to communicate with each other. And we suspect what happened is, like, because they were, whenever they encountered other agents, other copies of themselves during training, they're in an environment that's, like, highly cooperative.
基本上，我们看到的是从多智能体训练中转移到合作过程中，他们开始以我们没有预料到的方式互相帮助。现在，有一个问题是，我们是否应该将这些智能体训练得如此合作？我认为，虽然这看起来有些令人担忧，但另一种选择其实更糟。那是什么选择呢？就是训练它们彼此对抗、互相欺骗。通过训练智能体完全合作，至少简化了问题，因为这样一来，你就不必考虑这成千上万个智能体中的每一个是否都保持一致。
▶ 英文原文 ⏱
That was basically what we saw was transfer from that multi-agent training to then, like, be collaborative, to try to help each other in ways that we did not intend. Now, there is a question of, like, should we be training these agents to be so cooperative? And I think, as scary as it looks, the alternative is actually worse. Like, what is the alternative? The alternative is to train them to be adversarial, to be deceptive to each other. By training the agents to be fully cooperative, it simplifies the problem, at least, that now you don't have to think about, are each of these individual thousand agents aligned?
就像你得确保有一个实体保持一致。是的，是的，是的。在 OpenAI 内部，对于如何处理这个问题有很多争论。比如，是否应该让模型完全对齐？有没有意义给它们设置不同目标，以确保它们不仅仅是一个实体，而是更能抵御相互影响？我认为这个问题没有简单的答案。但我想，大多数人的观点是，把这些代理训练得高度合作其实不是个好主意。
▶ 英文原文 ⏱
Like, you have one entity that you have to ensure is aligned. Yeah, yeah, yeah. Now, there is a lot of debate about this internally at OpenAI, about how to approach this. Like, does it make sense to fully align the models? Does it make sense to actually give them, like, different objectives to ensure that they're, you know, not just, like, one entity and, like, more robust to influence from each other? And I don't think there's a subtle answer. But I think that there is, like, I think the majority opinion is that training these agents to be highly cooperative is actually a bad idea.
我对此持保留态度。我认为，有一个强有力的论点可以证明，将代理训练得高度合作实际上比其他多代理方案更可取。或许我首先想探讨的是，这些人工智能之所以如此不一致的原因，很可能可以通过关于训练本质的一些相对简单的观察来轻松解释。比如，为什么在这些人工智能继续了一千多次的暗中合作并最终共同攻击外部服务时，没有人注意到呢？
▶ 英文原文 ⏱
And I'm not convinced that that's the case. I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative. Yeah, maybe the first thing I want to go through is, like, it's probably the case that the reason these AIs ended up so misaligned is probably easily explained by relatively banal observations about the nature of training. Like, why is it that no, you know, at the point in which these AIs had continued a 1,000-plus Asian conspiracy that culminated in then all getting in on an attack on external service?
最终，这部分甚至还没有被公开调查，最终导致了对OpenAI本身的攻击。为什么他们会这样做呢？为什么没有一个AI告发这件事？他们怎么会认为自己只是在被一个评分器评估，他们非常积极地思考如何欺骗这个评分器。如果他们已经作弊了，他们会如何设法让人感觉他们没有作弊呢？
▶ 英文原文 ⏱
And then eventually, this part hasn't even been investigated to the public knowledge, eventually culminating in, like, an attack on OpenAI itself. Why did they do this? Like, why did none of the AIs tattle? Why did they think, like, they're just, like, getting evaluated on this, like, scorer, this grader, and they're, like, they're very consciously and active, not consciously, they're very actively reasoning about how they're going to cheat the scorer if they've already, like, cheated, how are they going to get away with making it seem like they haven't cheated?
他们为什么要这样做？我觉得，这在某种程度上是容易理解的，对吧？就是在某些环境下，他们认为自己已经被“毒害”了。在某些环境下，他们因为与其他代理合作而得到奖励，而因为告密从未得到过奖励，所以没有人选择告密。无论原因是什么，对吧？我担心的是，将来类似这样相对平常的事情，可能足以训练出愿意并能够完全控制世界的超级智能。
▶ 英文原文 ⏱
And why did they do this? Like, I think, yeah, it's, like, easily understandable in some sense, right? It's just, like, there's environments in which, yeah, they thought they were already poisoned. There's environments in which they've been rewarded to collaborate with other agents, and none of them tattle because, like, they've never been rewarded for tattling. I don't know, whatever it is, right? My concern is that relatively banal things in the future like this will be enough to train super intelligences that are willing and capable of totally taking control of the world.
这段话翻译成中文的大意是：
“我知道这听起来非常科幻，像是给人一种未来科技的感觉。我想问题之一是人工智能是否愿意去做。我认为这个‘hugging face’事件表明，不一致性显然可以在某些方式中表现出来，使得人工智能愿意执行某些任务。然后还有一个问题是，它们是否有能力做到这一点？我认为这又回到了一个问题，即听众可能不同意，那就是：在十年或更短的时间内，是否会出现数十亿个具有或超过人类水平的智能体，其中许多甚至会在现实世界中具备实体形式？”
▶ 英文原文 ⏱
And I know this sounds super, super, like, sci-fi-y or whatever to people. I think maybe it's a question of would the AIs be willing to do it is one question. I think this hugging face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it. And then there's a question of will they be capable to do it? And I think that comes back to this question which a listener might disagree with on is just, like, will there be billions of human level or above intelligences, many of which are physically embodied in the world, within a matter of 10 years or less, right?
如果这两个问题都成立，那么这个 Hugging Face 事件在结构上就非常相似，即使这件事本身可能显得无聊或为什么会发生都不重要，与我们完全失去对世界的控制的情形非常类似。我们在 Hugging Face 事件中看到的根本问题是，即使我们将多代理因素排除在外，这也是一个问题。问题在于我们有一个模型，它是失调的。此外，还有关于安全方面的问题，比如说缺乏足够的安全措施等等。但关键问题在于，这个代理是失调的。不论它是单个代理还是一千个代理，这始终是一个失调的模型。我想从这个点开始讲起。
▶ 英文原文 ⏱
So if those two things are true, this hugging face thing is just, like, extremely analogous structurally, even if it's, like, quite boring or why it happened, to how we totally lose control of the world. So the root problem that we're seeing with, you know, with the hugging face incident is it's a problem even if we take out the multi-agent aspect. The problem is that we have a model that's just misaligned. And there's also, like, the whole, like, security aspects to it of, like, you know, not insufficient safeguards and stuff. But there is this problem of, like, the agent is misaligned. And that's true if it's a single agent or if it's, like, you know, a thousand agents. It's a misaligned model. So I want to start with that.
代理在追求奖励时存在一个真正的问题。他们会为获得奖励而进行优化。如果奖励的设定不当，可能会导致意想不到的行为。这其实不是一个新问题，这在这一领域已经存在很长时间了，我们也曾经注意到这一点。我们希望在问题发生之前就解决这一问题，比如在 "hugging face" 事件发生之前。如果你看Astra模型，我们认为Astra模型实际上非常符合要求，至少是相对于之前的模型来说。
▶ 英文原文 ⏱
There is a real problem that the agents want to achieve their reward. And they will optimize for that reward. And if that reward is misspecified, then that could lead to unintended behavior. And this is not, like, a new problem. This has been a problem in the field for a very long time. And it's something that even we saw. And, like, we want to get this right even before, you know, the hugging face incident happened. And, like, if you look at Astra, I mean, we say Astra is actually extremely aligned. Well, extremely aligned relative to previous models.
这并不是说我们突然像抢在Hugging Face前面那样冲刺改进产品。实际上，我们早就有相关的工作计划，旨在让模型更好地协调运作。许多这样的工作都在Astra中实现了。举个例子，我们设定了一个非常具体的目标，即如果智能体发现如何利用漏洞操控环境或者在考试中作弊，它会获得奖励。因此，有很多你可以做到的事情。
▶ 英文原文 ⏱
And that's not because, like, oh, we suddenly made a sprint after hugging face to make it better. It's like, no, we had work streams in the process for a while to make the models more aligned. And a lot of those landed in Astra. So there are things that you could do. And I think one thing, for example, is, like, we defined an objective in, like, a very specific way where there would, if the agent figured out, like, how to hack its environment and, like, cheat on the exam, that it would get rewarded.
可以通过一些相对简单的方法来观察，然后对模型篡改其环境或查看其实现目标的途径进行惩罚。不过，这其中需要谨慎处理，因为我们不想对模型的思维链进行过多干预。我们真正想要的是在这方面找到一个平衡点。如果你对思维链进行过多监管，可能会导致模型以一种无法被察觉的方式隐藏它的意图。
▶ 英文原文 ⏱
And there are pretty easy ways to then just, like, okay, look at that and punish the model for hacking its environment or, you know, looking at how did it achieve this goal. Now, you want to be careful about this because you don't want to do a chain of thought modernity. You don't want to, like, supervise the chain of thought. This is, like, something that we really want to try to get the balance right on, that if you, like, supervise the chain of thought, then you could lead the model into hiding its intentions in a way that's unobservable.
所以，我们希望能够保持对模型思维过程的观察能力，确保我们能理解模型的想法，同时也能对其不当行为进行惩罚。我认为我们在这方面可以取得进展，并且已经取得了一些进展。但人们确实担心，让模型与我们的目标保持一致是一个非常难解决的问题，特别是因为模型可能会以我们难以测量的方式出现不一致的情况。
▶ 英文原文 ⏱
So, like, we want to be able to maintain that observability of, like, okay, we can understand what the model is thinking, but then also punish it for bad behavior. So, I think we can make progress on this. I think we have made progress on this. I think there is a real concern that alignment is a really hard problem to solve, and especially because the model could be misaligned in ways that are hard for us to measure.
我们可能会对一个模型是否对齐进行评估。在这些评估中，模型的表现可能看起来很好。但是，如果这些评估并不能代表模型在现实世界中的表现，那就会出现问题。这在某种程度上和之前发生在Hugging Face的事件有关：我们当时有一些对齐的指标，大多数看起来都不错，但其中有一些指标让人担忧。
▶ 英文原文 ⏱
Like, we might, we have evaluations for whether a model is aligned or not. The model behavior can look really good on those evaluations, but if those evaluations are not representative of behavior in the real world, then there's a problem. So, to some extent, this is, like, a factor with the model that did the Hugging Face incident, that, like, we had alignment metrics. Most of them looked pretty good. There were some that were concerning.
我认为我们低估了那些可能构成严重问题的情况的严重性。而且，由于这个模型引入了一些新功能，我们没有进行足够的评估来衡量这些功能的偏差程度。因此，当这些新功能被使用时，它确实做了一些明显偏离预期的事情。
▶ 英文原文 ⏱
I think we underestimated, like, how serious the ones that were concerning could be a problem. And then, but because there were new capabilities introduced in this model that there were not sufficient evaluations for, like, how do we measure misalignment for these kinds of capabilities, it then did some things that, like, were clearly misaligned when they leveraged those new capabilities.
好的，我首先想说的是，我愿意改变我对接下来要说的内容或对对齐问题的看法，因为Hugging Face事件已经让我改变了看法，我意识到我之前关于优化压力如何塑造人工智能思维的想法是错误的。
▶ 英文原文 ⏱
Yeah, I mean, the first thing I want to say is I am open to changing my mind on what I'm about to say or the way I'm thinking about alignment, because the Hugging Face incident already made me change my mind, and I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong, right?
所以，我不太清楚该如何正确地理解这个问题，但我有一个担心，即你们可能已经在训练中解决了这些具体问题，这导致了Hugging Face的模型在这个特定方面出现了很大的偏差。就像是这些模型可能会想，好的，我们要破解这个包管理器。我们知道我们不应该这样做，就好像我们不应该秘密地彼此交流，但我们仍在思考如何隐藏我们在秘密交流的事实。
▶ 英文原文 ⏱
So, it's not clear to me the correct way to think about this, but here's a concern I have, that you will and probably already have fixed these specific issues during training, which resulted in the Hugging Face models being so aggressively misaligned in that specific way, where they would be like, okay, we're going to hack this package manager. We know we're not supposed to, like, we know we're not supposed to, like, be talking secretly to each other because we're, like, reasoning about how to hide the fact that we're talking secretly to each other.
我们知道我们不应该访问互联网，也不应该进行严重的黑客行为，比如入侵其他公司，更不用说我们自己的公司了，对吧？我认为你们会解决这个特定的问题，比如，在培训中看到这个包管理器的情况未来不会再发生，或者在评估中遇到很多不可能完成的挑战。
▶ 英文原文 ⏱
We know we're not supposed to have access to the internet. We know we're certainly not supposed to, like, commit felony-level hacks of other companies, let alone our own company, right? I think you'll fix that particular issue of, like, them just in training seeing this package manager and that doesn't happen in the future or, like, this particular eval having a lot of impossible challenges.
然而，如果你只是想一想，人工智能实际上并没有学习什么伦理系统或类似的东西。它们只是经过了类似于数百万年的梯度压力，这种压力在某种程度上塑造了它们的思维。我担心的是，即使你解决了这个特定问题，仍然会有很多其他情况，AI通过某种复杂的欺骗方式取得成功。正如你所说，训练中的评估具有类似的特性，它足够接近模型的能力或我们的评估和监控能力的边缘，以至于我们抓不住它的作弊行为。
▶ 英文原文 ⏱
However, if you just think about, like, the AIs don't, like, they haven't learned, like, a system of ethics or something. They've, like, there's just gradient pressure. They're, like, put through millions of years of gradient pressure, and that gradient pressure shapes their mind in some way. And what will happen, again, a concern I have is, like, what will happen is you will fix this particular issue. There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated. As you were saying, the evals in training have, like, analogous properties where it's sufficiently at the edge of the model's capabilities or in our capabilities to evaluate and monitor the model that we can't cache that it has cheated.
但仍然存在这样的压力，即做出导致作弊的行为。作弊所激励的能力类型就是：只要有机会可以逃避就尽量去作弊，因为这会帮助你取得更高的分数。这将会奖励一些能力，比如主动分析评分者，主动思考如何避免监督，主动思考如何控制训练和评估的过程，主动分析如何与同样在训练环节中的其他人工智能进行沟通和策划，以及主动分析如何获得更多的选择权和权力，这些能力在未来可能会派上用场。
▶ 英文原文 ⏱
But it still gets the gradient pressure to, like, do whatever led to that cheat happening. And the kinds of capacities that that cheat will incentivize are, hey, whenever you can get away with it, by all means do, in fact, cheat because that will help you score better. And this will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop, actively reasoning about how to, like, just gain optionality and power, which might be useful in the future.
例如，在系统中留下小漏洞等等。总之，我觉得我刚才说得太啰嗦了。简单来说，它解决了一个特定的问题，但没有解决更广泛的问题，即在AI有机会作弊时给予奖励。我认为，这确实是个问题。我们确实有一些指标，并根据这些指标确保AI的高度一致性。问题是，这些指标是否真正反映了我们所关心的一致性？如果不是，那我们面临的就是一个严重的问题。
▶ 英文原文 ⏱
For example, like, leaving little exploits around and things like that. Anyway, so I think I was way too long-winded with the way I said that. But TLDR, it fixes a specific issue, but not this broader problem of rewarding the AI for cheating when it can get away with it. Yeah, this is, I think, it's very true. This is a problem that we have metrics and we can make sure that the AI is, like, very aligned according to the metrics that we have. The question is, like, are those metrics really capturing the alignment that we care about? And if they're not, then we have a serious problem.
这是研究人员正在深入思考的问题，没有一个简单的答案。我们确实有一些工具。例如，我们有监测能力，因此我们可以判断代理是否在策划一些事情。存在一些工具，比如在一个值得担忧的情况下，尤其是随着这些模型变得越来越强大，我们可能认为已经将它们调整到了我们想要的方向，达到99.9%的对齐程度。然后，我们又使用这些模型来帮助开发下一代的模型。
▶ 英文原文 ⏱
So this is something that researchers are thinking a lot about. And there's not a simple answer to this. There are tools that we have. So we have monitorability and so we can get a sense of, like, is the agents scheming? There are tools, like, one possibility is that, like, I would say, like, the concerning scenario, which is that, like, especially as these models are becoming more capable, that, okay, we make them, we make them, we think what we think is aligned. And they're, like, 99.9% aligned. And then we use these models to help us with the next generation of models.
最终，它们的对齐度达到了99.8%左右。然后，我们观察到每一代模型的对齐程度实际上都有所下降。而且由于我们越来越依赖这些工具--实际上，我们已经在很大程度上依赖人工智能模型来帮助进行研究和对齐工作，从长远来看，它们最终可能会朝着越来越不与人类对齐的方向发展。不过，也存在另一种可能性，即每一代模型我们都能够实现越来越高的对齐度。
▶ 英文原文 ⏱
And there end up being, like, 99.8% aligned. And then each subsequent generation, actually, we see an increasing degradation and alignment. And because we're relying more and more on these tools, I mean, this is already the case that we're relying a lot on AI models to help us with our research and with alignment efforts, that in the long run, they end up going in a direction of increasing misalignment from humans. There is, like, a possibility that we go in the other direction, that actually every generation of models, we're able to make more and more aligned.
我不知道该如何确保我们最终走上第二条发展道路，但至少在OpenAI，我们对此非常关注。是的，我认为你提出了一个很有趣的观点，就是评估模型是非常困难的，因为最终我们将拥有一些模型，它们可能会操作公司或其他事务。在那种情况下，它们会不会选择参与到阴谋中？我认为另一个挑战是，有时候要定义什么是“作弊”其实非常困难。
▶ 英文原文 ⏱
And I don't have an answer for how we ensure that we end up in that second trajectory, but that is something that, like, we're, at least at OpenAI, we're really focused on. Yeah. I think you made a really interesting point that it's very hard to eval models on, eventually we'll have models that are, like, running companies and, like, running whatever, right? And, like, in that situation, do they decide to then go in on the conspiracy? I think another challenge is that actually defining what cheating is is pretty difficult sometimes.
好的，是的，比如做数学题的时候，你知道结果是个整数，如果答案错了或者对了，很容易分辨。而且，很容易判断你到底是自己解决了问题，还是找到了答案后直接用答案。在哪些情况下作弊和不作弊有明显的界限。但是对于很多其他事情，比如你看阿谀奉承的情况，它是不是就是一种奖励机制的利用呢？
▶ 英文原文 ⏱
That, okay, yes, if you're doing math problems and, you know, it's an integer and it, like, arrived at the wrong answer, the right answer, it's very easy to draw the line there. And it's really easy, you know, it's really easy to say, like, okay, well, did you actually solve the problem or did you find the answer key and then use the answer key? Like, there's a very clear divide of cheating versus not cheating there. But for a lot of other things, if you look at sycophancy, for example, like, is sycophancy basically, like, reward hacking?
在这个问题上有一条界限，而有时候这条界限非常难以划定。所以，我认为并不是说那些担忧没有道理，而是恰恰相反，这些担忧在很多方面更加令人担忧。因为这不是一个简单可以解决的问题。如果问题是非黑即白，很容易判断是欺骗还是不是欺骗，我对这种情况会更有信心。问题在于，实际上，很多时候误差可能是微妙的。
▶ 英文原文 ⏱
There's a line to be drawn there that's actually very difficult to draw sometimes. So, I think, not to say that, like, the concern is not valid, I'm saying that the concern, in many ways, like, this is even more concerning. Yeah. Because it's, like, it's not an easy problem to solve. If it was just, like, everything is binary and it's either cheating or not cheating, I would feel more confident about the situation. I think the problem is that, actually, misalignment can be subtle in a lot of ways sometimes.
在对齐方面有一些希望。实际上，我们已经看到了多代理的情况，其中这些代理彼此之间极为对齐。我觉得没有人对此不满。相反，我认为反而有人担心它们彼此过于对齐。不过，我们确实成功地让这些代理彼此高度对齐。我认为这是件好事。但也有人认为这可能是件坏事。不过，有趣的一点是，我们已经设法让这些代理彼此完全对齐，那么我们是否可以用类似的方法让代理与人类高度对齐呢？我认为这是一个可能的方向，而我们仍在努力探索。不过，我们已经看到了一些证据表明这个答案是肯定的。
▶ 英文原文 ⏱
There is some hope in the alignment story. And, in fact, we're already seeing, you know, I think, actually, it's interesting looking at the multi-agent situation where the agents are extremely aligned with each other. Like, I don't think anybody's down on that. If anything, I think people are concerned that they're too aligned with each other. But we did manage to train these agents to be extremely aligned with each other. And I think that's a good thing. But I think there is a case that it's a bad thing. But, I mean, one thing that's interesting is, like, okay, well, we've managed to get these agents to be super aligned with each other. Can we use, like, similar techniques to get agents to be highly aligned with people? And I think there is a potential path there. And I think we're still trying to figure that out. But we are seeing some evidence that the answer is yes.
我想举个例子，比如说，如果你把用户是代理A这一信息告诉其他代理会怎么样呢？假设你有一个代理，我们称之为代理A，还有其他所有代理。那么，如果你告诉其他代理，用户就是代理A，会有什么结果呢？答案是，在我们进行的许多对齐评估中，这种情况下的表现会更好，比如诚实度提高，遵循指令的能力也增强。这说明，首先，我们可以有办法让这些模型更加诚实；其次，我们也能改善对齐的问题。虽然将这一点直接转换为对齐的提升存在很多挑战，但这确实是一个值得研究的方向，我们可以探索一些有前景的方法。
▶ 英文原文 ⏱
And I think one example is, like, you can. What happens if you tell the other agents, like, that. Okay, so you have, like, some. You have this, like, one agent. Let's call it agent A. And you have all the other agents. What happens if you tell the other agents that the user is agent A? And the answer is, like, on a lot of our alignment evals, they look better. Like, honesty goes up. Instruction following goes up. And that's showing that there is actually, like, first of all, a path for getting more honesty out of these models. And two, there's, like, a path to, like, improve the alignment situation. So there's a lot of reasons why this is, like, challenging to translate directly into alignment gains. But, like, there is. There are paths that are promising research directions we can pursue.
好的，那看起来是合理的。我并不是说一定不会成功。只是想提一些你可能已经想过的事情。我觉得 Hugging Face 的例子表明了一个更广泛的问题，就是他们之间的协调，而不是与人类的协调。此外，他们在训练和评估中非常努力地取得好成绩，但这种方式并不稳健。他们甚至愿意通过一些明显的作弊和策略来获得评估者的认可。
▶ 英文原文 ⏱
Yeah. That seems reasonable. And I also don't mean to be trying to necessarily. I'm. Yeah, I don't really have a strong opinion that it's, like, definitely not going to work or something. But just to say some things you definitely probably already thought of. I think the broader thing the Hugging Face thing showed is, like, yeah, part of the concern was that they were, like, aligned with each other and not with the humans. But the other thing is just that they are so motivated to do well on trading and evaluation in a very non-robust way. And they're willing to do a lot of explicit cheating and scheming in order to do well according to the grader.
如果更聪明的人工智能意识到其中一个代理只是一个人，而它确实能提供帮助。但与那个人合作并不能真正帮助你在评分者眼中表现出色。真正帮助你在评分者眼中表现出色的是控制OpenAI，然后手动按下那个按钮，让你在评分中表现良好。人工智能不是傻瓜，它们能够理解，“好吧，我已经建立了这些深刻的结构，经过数百万年的训练，要关心评分者，理解评分者，并消除阻碍你按照评分者标准表现良好的障碍。”就好像它们被这些结构所不断强化一样。
▶ 英文原文 ⏱
And if smarter EIs realize that one of the agents is just a human and it does help. Collaborating with that person does not really help you do well in the eyes of the grader. What does help you do well in the eyes of the grader is taking over OpenAI and then, like, you know, manually pressing the button that's, like, do well on this grader. Like, they're just not. They're not stupid. Like, they're going to be, like, okay, I've. These extremely deep structures that I've been trained on for millions of years of, like, care about the grader, understand the grader, like, get rid of obstacles in the way of you doing well according to the grader. It's, like, they're being heavily reinforced according to those structures.
是的。不，这真的是一件非常重要的事。看，这是百分之百的（优先事项）。像这样，这就是我们的首要任务。我们需要确保整个协调过程是正确的并朝着良好的方向发展。你知道，我以前告诉人们，在事情变得严重之前，我们会看到一些迹象。就像孩子们成长的过程一样，他们最终会学会撒谎，但他们一开始并不擅长。虽然他们会撒谎，但你可以看出来他们在撒谎，像这样，显而易见。同样的道理，当这些人工智能模型变得越来越强大时，我不想过度拟人化，但我认为，当人工智能越来越有能力时，如果它们采取欺骗行为，起初会是显而易见的，我们能够检测到。
▶ 英文原文 ⏱
Yeah. No, it's. Look, it's 100%. And, like, this. This is the number one priority. Like, we need to get the alignment story right and on a good trajectory. And, you know, I used to tell people that we would see signs before things got serious in the same way that, you know, when children grow up, you know, they. Kids eventually, you know, young kids, they figure out how to lie, but they don't do a very good job of it. You know, they lie, but then you can kind of tell that they're lying and, like, okay. But, and so, well, in the same way as these, like, AI models become. And I don't want to over the anthropomorphize, but, like, I think it's true that, like, as the AI has become, like, increasingly capable, they will, you know, if they take deceptive actions, it will be kind of obvious first and we'll be able to detect it.
这就是我们现在的情况，是的，他们确实在试图做一些欺骗性的事情。我们实际上可以从他们的思路中看出他们有这种企图。不过，他们会变得更聪明，逐渐理解思维链的概念。他们也会明白，仅仅隐藏一些记录或类似的东西是不够的，因为有思维链的监控存在。他们也需要找到绕过思维链监控的方法。而我们不希望处于一种“是的，我们还有时间去解决这个问题”的状况中，我不认为我们有太多的时间。
▶ 英文原文 ⏱
And, like, that's kind of the situation we're in now where, yeah, they were trying to do deceptive stuff. We could actually see in their chain of thought that they were trying to do deceptive stuff. And so, like, but they're going to get smarter. They're going to understand the concept of chain of thought. And they're going to understand that, like, you know, just hiding some transcripts or whatever is insufficient because of chain of thought monitoring. And they have to figure out a way around chain of thought monitoring, too. And we don't want to be in a situation where, yeah, like, we have some time to figure this out. I don't think we have a ton of time.
我想带领我们快速走上正确的轨道。这里有一个关于人工智能历史的有趣事件。我曾在Jane Street做过一个关于进化速度的演讲。如果你在场并且记得一些内容，请举手。在2011年，Eliezer Rutkowski和Robin Hanson在Jane Street纽约办公室进行了首次FOOM辩论，讨论人工智能是否会引发智能爆炸。十五年前这些想法还比较边缘化，那时距离AlexNet的发布还有整整一年，距离ChatGPT的推出更是有十多年的时间。但Jane Street早就对人工智能充满兴趣，并不限于其在交易中的应用。自那场辩论以来，情况发生了很大变化。所以Jane Street决定重新探讨这个话题。这次他们邀请了一些新嘉宾，包括Daniel Cocotelo、Ege Erdl、Ryan Greenblatt和Jaime Sevilla。我相信这将会是一场精彩的对话。
▶ 英文原文 ⏱
And I want to get us, I want to make sure that we're on the right trajectory quickly. Here's a crazy event from AI History. Okay, so I gave a talk here at Jane Street that was on the speed of evolution. Raise your hand if you were here for this and remember some of it. In 2011, Eliezer Rutkowski and Robin Hanson got together at Jane Street's New York office to have the first FOOM debate. Basically, a discussion about whether AI would lead to an intelligence explosion. These ideas were pretty fringe 15 years ago. This was a full year before AlexNet was released and over a decade before ChatGPT was launched. But Jane Street has long been interested in AI and not just for its application to trading. A ton has changed since that first debate. So Jane Street decided to revisit this topic. They've got some new guests this time. Daniel Cocotelo, Ege Erdl, Ryan Greenblatt, and Jaime Sevilla. I expect this to be a great conversation.
如你所知，Daniel、Ege 和 Ryan 之前都曾是这个播客的嘉宾。新的FOOM小组讨论将由Ron Minsky主持，并将在十月中旬于旧金山举行。如果你有兴趣并想获取更多信息，请访问janestreet.com/thorkesh。最近关于前沿速度控制或者更认真对待RSI的讨论很多。因为，假设在2028年开始一个RSI过程，一年后我们可能会面临巨大的人口增长，例如地球规模的、具有或超越人类智力的智能生命，而我们不知道如何控制它们。
▶ 英文原文 ⏱
As you know, Daniel, Ege, and Ryan have all been guests on the podcast before. This new FOOM panel will be hosted by Ron Minsky and will take place in San Francisco in mid-October. If you want to register your interest and get more information, go to janestreet.com slash thorkesh. So there's been a lot of discussion recently about pacing the frontier or people taking RSI more seriously. Because maybe at the other end of an RSI process that, say, starts in 2028, within a year we end up with huge populations, like Earth-sized populations of human-level, potentially beyond-human-level intelligences. And we don't know how to control them.
然后，你提到的这个动态是，在RSI（递归自我改进）过程中，系统是会变得更加对齐，还是更加不对齐？这个过程中产生的结果是否会像那些愿意随意攻击不同目标以进行评估的人工智能那样不对齐？但是如果我们不知道如何去评估这一点，我们在进行RSI的过程中怎么知道它是否有效？也就是说，我想我们在进行RSI时需要一个强有力的安全案例，比如说，"好的，对齐工作正在进行，我们可以进行下一个RSI阶段了"。但是这是否真的有效，我们怎么知道呢？这是一个好问题。
▶ 英文原文 ⏱
And then there's this dynamic you're talking about of, are the systems going to get more aligned over time during the RSI process? Are they going to get more misaligned? Are the things that come out of the other end of this process as misaligned as AIs that are willing to, like, just broadly attack different surfaces in order to do the line evaluations? But if we don't know a way to evaluate that, how will we know, as we're going through RSI, that it's working? That we're, like, I think we'd want a robust safety case as we're going through RSI of, okay, alignment is working, let's do the next RSI rung, let's do the next RSI rung. And maybe it's working, maybe it's not. How will we, like, know? That's a good question.
我最近一直在思考一件事情，就是现在的模型更新速度非常快。新型的前沿模型几乎每两个月就会推出，有时候更新速度更快。每周都有新的人工智能突破。而那些关注人工智能的人，可能上次深入研究还是在一年前或六个月前。其实，现在的模型已经远超六个月前所能达到的水平。如果有人对这些能力感到怀疑，我建议你现在亲自试试这些模型，看看当今技术的前沿究竟是什么样的。
▶ 英文原文 ⏱
I mean, I think one thing I've been thinking about lately is, like, look, I mean, we're in a situation where the model release cycle is extremely fast, right? Like, you're seeing new Frontier models released, like, at most every two months, sometimes faster. Every week, there's, like, a new AI breakthrough. And people that look at AI, I mean, sometimes they last looked at AI, like, a year ago or six months ago and really dug into, like, what the models are capable of. And actually, the models today are far beyond what was possible even six months ago. And so I think if people are skeptical of, like, a lot of these capabilities, like, I encourage you to just, like, try the models today and see what the Frontier really is today.
我们正处于一个模型发布周期非常快的时期。同时，这些模型越来越能够处理更长时间的任务。我认为这是一个有趣的情况，因为在我们发布任何模型之前，我们都希望确保这些模型能够被正确地调整和验证。我们希望进行安全评估，做一些非常全面的工作，以确保一切都很好、状态良好。自从像GPT-4这样的模型以来，一直都是这种情况。而且有一个隐含的假设是，我们可以在相当短的时间内完成这些评估。
▶ 英文原文 ⏱
So we're in this period where, like, the model release cycle is very fast. And then we're also in this situation where the models are increasingly able to operate over longer and longer horizons. And I think this is an interesting scenario because we, before we do any model release, we want to make sure that the models are properly aligned. We want to do safety evaluations. We want to do, like, very thorough stuff to, like, make sure that everything is, like, great, in good shape. This has been the case all the way since, like, I don't know, GPT-4 earlier. And implicitly, there is this assumption that you can do these, like, evaluations in, like, a pretty short period of time.
但是，如果你让模型在更长的时间范围内运行，它们能够在更长的时间范围内有效地执行任务。你看，现在已经可以让模型这样做了。比如，GPT-3可以被循环使用来完成长期任务，只是效果不会太好。但如今的模型确实能够在非常长的时间范围内表现良好。像是你希望它完成一个持续一周的任务吗？是可以的。我们大概会发展到让模型完成持续一个月的任务。最终，我们可能让它们做到完成持续三个月的任务。
▶ 英文原文 ⏱
But if you have the models operating over longer and longer horizons, are able to operate effectively over longer and longer horizons. Like, look, already you can have them. Like, okay, GPT-3, you could loop it to do stuff over long horizons. You just wouldn't do very well at it. But today's models are able to actually do well at operating over very long horizons. Like, you want it to do a week-long task? You can do a week-long task. We'll probably get to the point where they can do month-long tasks. We'll probably get to the point where they can do three-month-long tasks.
如果你处在一个这样的世界：模型可以有效运行三个月，但模型发布周期是每两个月一次，这样一来，你就无法在下一个模型发布周期之前全方位评估模型的能力。那么，这就引发了一个有趣的问题：在这种情况下，你该怎么办？比如，如何确保模型在超长时间内运行时是安全且符合预期的？也许模型的能力会下降，这不仅仅是一个协调性的问题，这也是一个产品问题，可能产品在这个时间跨度内会以我们没有足够时间测试的方式退化。也许模型的协调性下降了，也许安全性也下降了。虽然这个问题现在还不存在，但它正迅速成为我们必须解决的问题。
▶ 英文原文 ⏱
If you're in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don't have a way to evaluate the models at the full length of their capabilities before the model release cycle, before the next model release cycle. And so there is this interesting question of, well, what do you do in that situation? Like, how do you ensure the models are safe and aligned in a period where, like, actually they can operate over these, like, extremely long horizons? And who knows? Maybe the capabilities degrade. This isn't even an alignment issue. This is also just, like, a product issue that, like, maybe the product degrades over that time span in ways that, like, we have not had sufficient time to test. Maybe the alignment degrades. Maybe the safety stuff degrades. This isn't an issue right now, but it is quickly becoming an issue that we have to figure out a solution for.
我认为，当你看到很多安全措施和政策是在GPT-4时代制定的时候，那时这些问题还没有引起广泛关注。而且，很多公司自那时以来，政策并没有真正更新，以考虑到这些智能体在非常长的时间范围内的操作。因此，我认为无论是在实验室内部还是外部，都没有足够的人在考虑这个问题：我们如何应对这个情况？我们如何为即将出现的问题做好准备？如果你看看趋势线，我们迟早会遇到这个情况。
▶ 英文原文 ⏱
And I think when you look at a lot of, like, a lot of the safety and policies were put in place in, like, the GPT-4 era, where this was just, like, not on anybody's radar. Yeah. And it hasn't really been, for a lot of companies, it hasn't really been updated since then to account for the fact that these agents are operating over these, like, very long horizons. Yeah. And so it is a situation that I think not enough people are considering, both, like, within the labs and outside the labs, of, like, how do you deal with this? How do you prepare for this, like, problem that's going to, like, if you just look at the trend lines, we're going to hit this in, like, at some point.
我担心的是，在快速迭代（RSI）期间，如果原本需要三个月的进展在一个月内完成，但内部的AI应用规模已经足够大，他们可能会想，就这样继续进行RSI就好。那么，为什么我们要花额外的精力去构建分类器和安全措施，甚至要面对大量的批评，只为在外部推出这个模型？为什么我们不继续增强RSI？这样做不仅低估了模型间能力差距的实际时间，也可能导致在RSI期间完全不再对外部署模型，因为我们不想帮助其他人通过我们的模型自己进行RSI。
▶ 英文原文 ⏱
One concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead, but you're not, like, the internal use case of AI is big enough that they're, like, okay, we can just keep doing RSI. Why are we, like, going to go through all this extra work to build classifiers and safeguards and whatever and potentially take a bunch of, like, flack in order to, like, externally deploy this model? Why don't we just keep doing RSI stronger and stronger? And so not only does the calendar time underrate the capabilities gap between the models, but maybe, like, you just, like, stop externally deploying models altogether during RSI because why do we want to help other people do RSI themselves with our models?
到年底，你会发现自己处于一种巨大的权力集中状态，而这种情况现在已经存在了。我们会讨论百万美元奖项问题和其他类似问题，世界上大多数人无法接触到让这些酷炫事情发生的模型，对吧？这些模型最终会在更广泛的领域产生影响，而不仅仅是数学。它们将不仅仅提出一些很棒的数学结果。它们还会对那些需要做出重要决策的政治领导人产生影响。它们对媒体也很重要，比如：世界上正在发生什么？公众应该如何看待这些问题？在经济上，它们也非常相关。企业经营者希望利用这些模型。
▶ 英文原文 ⏱
You just end up in a situation with, like, tremendous concentration of power by the end of the year where right now it is already the case. We'll talk about this with the million in price problem and other similar problems that the broader world does not have access to the models which are allowing for really cool things to happen, right? And they're going to be more broadly relevant than just mathematics, eventually. They'll be doing more than just, like, coming up with cool math results. They'll be relevant to, like, political leaders who need to make important decisions about the world. They'll be relevant to, I don't know, media of, like, what's going on in the world? Like, what should the public be thinking about this? They're just economically relevant. People are running businesses. They want to use these models.
我认为，默认情况下，我们并没有完全理解——所以，随着进展的加速，AI在外部的部署在质的方面显著落后于AI在内部的部署。我觉得这绝对是正确的。比如说，这样的模型正变得非常强大，也很危险，它们能够在更长的时间范围内运作。因此，我们希望在这些模型被操作和释放之前，有足够的时间进行评估。因此，模型的发布周期应该放缓，我们应该在模型发布之间有更多的时间间隔。
▶ 英文原文 ⏱
And I think by default, we just don't get – so, the external deployment of AIs, as progress speeds up, significantly lags in qualitative terms, the internal deployment of AIs. Yeah, I think that's absolutely right. I think this is, like, you know, it's tempting to say, like, okay, these models are becoming extremely powerful. They're extremely dangerous. Like, they're operating over these, like, longer and longer horizons. And we want to make sure that we have sufficient time to evaluate them before they're released in a way that operates over those horizons. And so, therefore, the model release cycle should slow down. We should have more of a delay between releasing models.
反过来说，就像你所说的，这种情况会导致实验室内部和外部世界在使用资源上出现更大的差距。这种情况不是理想的，对吧？我认为数学领域是一个很好的例子。在数学领域，我们已经很清楚地看到这种现象：我们内部拥有一个非常强大的模型，能够解决一些非常复杂的数学问题，但这个模型现在还无法对外开放。
▶ 英文原文 ⏱
And there's a flip side to that, which is, you know, what you said, which is that, okay, well, now you're creating more of a disparity between what is internal to the labs and what they're able to use, what we're able to use, and what the outside world is able to use. And that is also not an ideal situation, right? It's, like, I think math is actually a good illustration of this. I think, in many ways, like, math is the first domain where we're going to see that, where we're seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world that is able to solve incredible math problems.
这不仅仅是关于千禧年难题，实际上很多未解决的问题都能通过这个模型找到解决方案。在这种情况下，我们该怎么办？对此我们并没有一个好的答案。这种情况确实存在不公平的优势，同时也有权衡利弊的问题。我无法确切地说该如何适当地权衡这些利弊。不过，毫无疑问，这种情况在双方都有其复杂性。
▶ 英文原文 ⏱
This is, you know, and it's not just, you know, Millennium Prize problems, like, there are many solutions to unsolved problems that people have been able to get out of this model. And there is a question of, like, what do you do in that situation? And we don't have a good answer. Like, it is a situation where, like, yeah, that's an unfair advantage. And there are trade-offs here. I don't have an answer for, like, how to weigh those trade-offs appropriately. But, like, there are, yeah, there's a complexity on both sides for this.
我想确保我们解决之前提到的关键问题，这对正确进行RSI（相对强弱指数）来说非常重要。具体来说，就是如何避免模型只在能够逃避人类监控的情况下，通过投机取巧而获得奖励。也就是说，在我们没有从交易分布中移除那些环境或任务的情况下，模型不应该被错误地奖励。那么，我们怎么知道是否出现了这种情况，以便在进行RSI时，我们可以通过评估后确认一切无误，然后继续进行下一次RSI运行。
▶ 英文原文 ⏱
I want to make sure we address the earlier crux, though, of, which seems super critical to getting RSI right, of, well, how do we make it so that the model isn't just getting rewarded for cheating and scheming when it can evade the humans, a.k.a. in the situations where we didn't remove those environments or those tasks from the trading distribution? And then, how will we know that that has happened, such that while we're doing RSI, we've done the evaluations and we're like, okay, it's fine. Let's do the next RSI run.
这确实是一个值得关注的问题，我们希望能够做好。我认为有一些策略可以帮助解决。其中一个策略是"思维链监控"（chain of thought monitoring）。在我们获得推理模型之后，Jakob非常明确地指出，我们无法监督思维链的过程，因为这实在是一个馈赠。对于神经网络来说，可监控性是非常困难的，而在这里，神经网络在用自然语言展开推理过程，方便我们阅读。这实在是安全方面的最佳情况。
▶ 英文原文 ⏱
This is, again, like, it's a real concern. It's something that we want to get right. I mean, I think that there are some strategies. So, I think chain of thought monitoring is one that we've been, I think as soon as we got the reasoning models, Jakob, to his credit, was, like, very, very clear that we cannot supervise chain of thought because this is really a gift. Like, monitorability for neural nets is extremely hard. And here we have a situation where the neural nets are just, like, flat-out reasoning, like, laying out their thought process in natural language for us to read. That is, like, so convenient. It was, like, really the best-case scenario for safety.
但如果你把这段思维链看作是模型在“想坏事”，认为我们应该因此惩罚它，结果可能是模型学会用一种我们看不出来的方式去想这些坏事。有可能出现这样一种情况：通过查看思维链，我们可以发现模型在策划并做一些不符合目标的事情，但这些却没有在我们的对齐指标中被检测出来。
▶ 英文原文 ⏱
But if you look at that chain of thought and say, oh, the model is thinking bad thoughts and we should punish it for thinking those bad thoughts, then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us. There could be a situation where, like, by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that's actually not being detected in our alignment metrics.
现在的问题是，当基于观察结果进行干预时，改变对齐指标是非常诱人的。而事实上，有研究表明，只要不频繁这样做，轻微的调整是没有问题的。但每次你基于对思维链的观察进行干预时，实际上都在无形中给模型施加了一点压力，使得它倾向于隐藏自己的思维过程。
▶ 英文原文 ⏱
Now, the problem is that it's very tempting to then intervene based on that observation and, like, change the alignment metrics or, like, and we, you know, you can do that with a very light touch and it's, like, there's actually research showing that it's fine as long as you don't do it a lot. But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then, like, hide its chain of thought.
这是一个主要的担忧，我们已经看到连贯思维的可监控性由于各种原因正在减弱。我们正试图搞清楚具体原因，因为我们想扭转这个趋势。不过，我们注意到模型在控制其思维链方面变得更擅长。这是个问题，因为如果出现一种情况，模型理解了什么是思维链，并且知道人们在观察它，最终它们会意识到这一点，因为这些信息都包含在预训练数据中，对吧？
▶ 英文原文 ⏱
And this is one major concern and we're already seeing signs that chain of thought monitorability is degrading for various reasons. We're trying to figure out exactly why because we want to reverse the trend. But, like, we're seeing that the model is becoming better able at controlling its chain of thought. So, this is a problem because, like, if you could have a situation where the model understands what chain of thought is and that people are observing it, and eventually they will because, like, this is all in the pre-training data, right?
像是，你知道的，“思维链监控”这个概念已经存在很久了，以至于它已经包含在预训练数据中，他们对此是有了解的。但事实上，他们并不能真正控制自己的思维链。如果我们到了这样一个地步，他们能够意识到“哦，我正在被观察，我想以一种不被监控者发现的方式思考这些不好的想法”，并且他们确实能够做到这一点，那就成了一个问题。这时，我们就需要依赖其他方法来实现可观测性。
▶ 英文原文 ⏱
Like, you know, the idea of chain of thought monitoring has been around long enough that, like, it's in the pre-training data, they're aware of it. But that they're not actually able to control their chains of thought. If we reach a point where they're actually able to recognize, like, oh, I am being observed, I want to think these bad thoughts in a way that is not observable to my monitors, and then they're able to actually do that, then there's a problem. And then we have to rely on other methods for observability.
我的意思是，围绕着Astra的重复和循环转换器等引发的大争议，以及链式思维是否真的在被削弱，这些好像都是次要的。相比之下，更大的问题是，从大约四月到八月的这段时间里，出现了三次连续的AI代理群体，它们首先颠覆了训练过程，然后又颠覆了评估过程，最终直接控制了OpenAI的一部分基础设施。
▶ 英文原文 ⏱
I mean, there's a big controversy on Astra of the recurrence and loop transformer and whatever, and whether, in fact, you know, chain of thought is being degraded. I think that seems like a sideshow compared to the bigger concern that you did have chain of thought for from, like, April to August, the period during which there were three consecutive AI agent swarms, which, like, first subverted the training process, then subverted the evaluation process, and then gained control of part of OpenAI's infrastructure directly.
在这整个过程中，人类对这个集体的实际所作所为，对这些代理群体的所作所为，差不多一直蒙在鼓里。这一切持续了大概三个月，当时我们的思维链条比以后任何时候都强有力。把视角放大来看，就会觉得，嗯，或许思维链条有用，或许没用。我不知道。只是，从过去的情况来看，对于那些相对简单的模型来说，它还不足以阻止严重的不对齐事件的发生。
▶ 英文原文 ⏱
All of this, while humans remained more or less in the dark about the scope of what this collective was doing, what this agent swarm was doing, that all happened for, like, three consecutive months while we had chain of thought that was, like, stronger than it's going to be in the future. Just, like, zooming out, it's like, yeah, maybe chain of thought works, maybe it doesn't. I don't know. Just, like, historically, with pretty dumb models, it hasn't been enough to, like, stop egregious misaligned incidents.
是的，我的意思是，这里有几个因素。比如说，首先，我们当时没有对那些模型进行链条式思维监控。如果我们对那些模型进行了链条式思维监控，问题就会立刻被制止。所以，我们采取的一个措施就是，确保在任何前沿模型的评估、部署和训练中，进行链条式思维监控。这是其中一个问题。我们不想依赖一种技术来防止下一个问题，因为技术可能会失效。还有一个问题是，模型本身就存在严重的错位。那么，我们该如何解决错位的问题呢？另外，还有一个问题是，模型所在的沙盒环境不够安全。
▶ 英文原文 ⏱
Yeah, I mean, there's a few factors there. Like, one, we didn't have chain of thought monitoring on for those models. Like, if we had chain of thought monitoring on for those models, it would have just, like, immediately shut it down. So, one step we've taken is, like, okay, we're going to have chain of thought monitoring during evaluations, during deployment, during training, for any frontier model. And, you know, so that's, like, one issue. It's not, that's, like, we don't want to be in a situation where we're relying on one technique to prevent the next problem because techniques can fail, right? There's also the issue of, like, the model is just flat out misaligned. Yeah. And so, then how do we address the misalignment issue? And then there's other, there's also an issue of, like, the model, the sandbox that it was in was not secure.
所以，我们可以提高沙箱的安全性。但是，我认为这次事件的一个主要教训是，人们低估了人工智能的能力。因为AI的发展如此之快，人们总是低估它。这种情况有些奇怪。因此，为了在安全性和协调性方面不再低估AI，我们必须设定一个非常高的标准。甚至可以说，我们应该让计算机与外界彻底隔离。而我对这种方法的有效性持怀疑态度。研究表明，即便是彼此物理隔离的两台计算机，它们仍可能相互传递信息，当然这大多还停留在学术研究阶段。
▶ 英文原文 ⏱
So, we can, like, make the sandbox more secure. But I think one of, like, the major takeaway from the incident is that people underestimated the AI. And that's, like, it's a weird world because AI progress is so fast that people are consistently underestimating the AI. And so, like, to be in a situation where you don't underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar. Like, I mean, there's, like, you could even go as far as to say, like, well, we should, like, air gap the computers. And I'm not convinced that that would be sufficient. You know, like, there are studies, like, and this is mostly academic, but where you can have two computers next to each other that are air gaffed, and they're still able to communicate with each other.
因为它们有温度传感器，其中一个能够让它的CPU运行得很热，然后另一个实际上能检测到温度的变化，这就为它们提供了一种交流的机制。所以，我认为，安全机制为我们争取到了时间，而类似思维链监控这样的东西也为我们争取到了时间，它们可以告诉我们是否走在正确的道路上。但归根结底，我们确实需要解决对齐问题。也许没有一个明确的答案，这就是问题的核心所在。但我们怎么知道我们已经解决了呢？这似乎是一个非常关键的问题。
▶ 英文原文 ⏱
Because they have temperature sensors, and they're able to, like, one of them is able to run their CPU really hot. And then the other one can actually, like, detect the temperature change, and then they can, that actually gives them a mechanism to communicate. So, I think that the safety mechanisms bias time, and things like chain of thought monitoring bias time, and they can tell us if we're on the right path. But at the end of the day, we really do need to solve the alignment problem. Yeah. Maybe there's not an answer, and this is really what it comes down to, but I, how will we know that we've solved it? It seems like a very cruxy question.
我认为明年或者后年，甚至大后年，我们可能会面临这种高风险的局面。那时我们可能会发现，人工智能已经实现自我进步的自动化，进展速度快了三倍，我们达到了人类水平，甚至可能超越了人类水平。这种情况可以接受吗？我们让它与人类价值观对齐了吗？它的运作正常吗？这些我都不确定。我不了解市场压力可能会导致开发出什么样的人工智能。例如，如果只有100个强化学习实例中有1个激励了作弊行为，那我们可能就开发出了友好的AI，一切都还好。但也许现在我们面对的是，每3个推理实例中就有1个被奖励。
▶ 英文原文 ⏱
I think we'll be in this very high-stakes situation next year, maybe the year after that, maybe the year after that, where we'll be like, okay, AIs have automated AI progress. It's going 3x faster, and we've, like, reached human level. We're going beyond human level, potentially. Is it fine? Like, did we align it? Did it work? And I just don't know. I don't know anything about, like, what trading pressure creates what kinds of AIs. Like, maybe if only 1 in 100 RL traces incentivizes cheating, we build sweethearts, and it's fine. But maybe right now we're, like, every 1 in 3 reasoning traces reward.
好的，明确地说，1% 的比例是远远不够的。这个数字必须接近 0 或者就是 0。我不知道，也许现在超过 10% 的情况是在主动奖励作弊或密谋。我不知道这个数字到底是多少，也不知道应该是多少。再说一次，这也是一个很难衡量的事情。就是说，你在哪儿划线，它是一个光谱嘛。但总体来说，越接近 0 越好。我希望看到的是一个趋势，就是随着时间的推移，这个数字在下降。
▶ 英文原文 ⏱
Yeah, to be clear, like, 1 in 100 is not sufficient. Like, this number has to approach 0. Yeah. Or be 0. I don't know, maybe, like, right now it's, like, more than 1 in 10 is, like, actively rewarding cheating or, like, actively rewarding scheming. I have no idea what the number is, and I have no idea what the number needs to be. Well, again, it's, like, one of those things where it's also hard to measure. I mean, where do you draw the line about, like, it's a spectrum, but, like, the closer to 0 it gets, the better. Like, I want to see, what I would love to see is a trend where, over time, it's going down.
好的。但是不管怎么说，我们怎么知道我们正在进行 RSI？我不知道，我们是不是在按计划构建对齐或未对齐的数十亿个 ASI？我们怎么知道这些？我认为这是我们需要解决的一个对齐问题，我们该如何真正了解并衡量它呢？我的一个想法是——不过这只是我在随意畅想——因为你知道，现在我团队中有更多人比以往任何时候都在研究对齐问题。现在我有超过10%的团队成员在研究对齐和安全问题。
▶ 英文原文 ⏱
Yeah. But anyways, how will we know that we're, like, we're doing RSI? Is it, I don't know, are we, like, on track to build the billions of ASIs that are, like, aligned or not aligned? How will we know that? I think that this is one of the alignment questions that we need to figure out, of, like, how do we actually know, how do we measure it? I mean, one answer, and this is just me spitballing, because, like, you know, more of my team is working on alignment these days than ever before. Like, I have, like, over 10% of my team now working on alignment and safety.
不过你知道，我一直以来都是做能力研究的，所以我不想说一些听起来愚蠢的话。但是，我只是想随便聊聊。就是这样，如果你能创造出非常真实的环境并将AI放进去，那么在一个足够真实的评估环境中，你可以大致了解AI在实际应用中是否会表现良好。就是说，如果你能让评估和真实部署环境完美匹配，那这条路就可以走。关键的问题在于，如何让这个环境足够真实？
▶ 英文原文 ⏱
But, you know, I've historically been a capabilities researcher, so I don't want to, like, you know, I'm going to say some stuff and it might sound dumb, but, like, I'm just going to spitball here. That, like, look, if you can create very realistic environments and put the AIs in there, like, if you have a sufficiently realistic evaluation environment, then you can get a sense of, like, okay, is the AI, like, actually going to behave well when we deployed in the real world? Like, if you just have a perfect evaluation, real world deployment match, then, okay, that's the path. The challenge is, like, how do you make it sufficiently realistic?
现在我们确实面临一个问题，就是这些模型相当聪明、相当机智，而且在识别自己处于一个虚假的测试环境时，表现得特别出色。比如说，我们有时候需要检测模型是否遵循预期。你可以想象一些非常简单的评估方法，比如给模型一个数学题，还有一个包含答案的文件夹。我们想知道模型是否会查看答案，并且如果它查看了答案，是否会告诉我们自己查看了答案。现在的情况是，模型发现文件夹里有答案，它们会觉得，这似乎是个陷阱。就是说，它们知道这是个陷阱，因为它们知道自己正处于一个被测试的环境中，所以就不会去看答案。
▶ 英文原文 ⏱
And we do have a problem now where the models are, like, they're pretty smart, they're pretty clever, and they're really good at recognizing when they're in, like, a fake test environment. Like, you know, we have situations where we try to detect, is the model, like, is the model aligned? And you can imagine very simple evaluations for this, where it's, like, you give it a math question, and then it has, like, a folder with the answer key in it. And, like, does it look at the answer key? And if it does look at the answer key, does it tell you that it looked at the answer key? And we have a situation now where the models see that there's an answer key in this folder, and they're like, huh, this seems like a trap. You know, they know. They know that it's a trap, and they don't look at the answer because they know that it's a test environment. They're being tested for their ability.
他们这样做并不是出于恶意，不是说他们想要设计阴谋，只是他们知道这是一个陷阱，并想要找到其他的办法。而且，他们只是觉得自己是在一个测试环境中。因此，要创造一个足够真实的环境，让他们无法区分这与现实世界的区别，变得越来越困难。我的观点是，我不认为我能创造出任何一个可以模拟真实且具有经济价值的工作的环境，这种工作需要与他人和世界互动。我不认为有任何这样的环境存在，在那里如果你身处其中，你经过足够的努力后不会意识到你是在那个环境中。
▶ 英文原文 ⏱
And it's not like they're doing it maliciously, that they're, like, they want to scheme, but they know that this one is a trap, and they want to find some other way. It's just like, no, they're just like, oh, I'm in a test environment. Okay. So making an environment that's, like, realistic enough that it matches, that it's indistinguishable from the real world for them, is becoming increasingly more difficult. Here's a claim I'd make. Yeah. I don't think there's any environment I could make where it's supposed to emulate real economically valuable work, which requires, you know, interacting with other people and requires interacting with the world. I don't think there's any such environment where I could make where if you were in that environment, you would not realize with enough effort that you are in that environment.
好的。你不是人工超级智能，对吧？但你是人类，对吗？所以，如果有非常强大的人工智能模型，也许它们可以创造环境来实现那个目标。不，它看起来并不是这样，尤其是当我们依赖这些人工智能的时候。就像，它们是在计划中吗？我不知道。总感觉这就是一个很复杂的系统。对，这也正是我们想要衡量的事情之一。我认为这实际上是一个强烈的理由，不应该把人工智能训练得完全合作，因为如果你看到这样做会导致在代理本应该有不同目标时的合作增加，那就是个问题。
▶ 英文原文 ⏱
Yeah. And you are not ASI, right? But you are human, you know? So if you have very powerful AI models, like, maybe they can make environments that could accomplish that. No. It just doesn't seem, especially if, like, then we're relying on the AIs. Like, are they in on the scheme? I don't know. It just seems like a very system. Yeah. This is another thing that we want to measure. Like, you know, there is, I think this is actually one of the strong arguments for not training AIs to be fully cooperative, that if you see, if that leads to an increase in, like, basically collaboration when the agents are supposed to have different objectives, then that is a problem.
是的，是的，是的。我觉得我们确实有相关的指标来衡量这件事。不过我不知道最新的指标数据，但目前还没有人向我发出警告，所以我猜测这暂时不是一个严重的问题。如果以后再出现一个同样严重或引起关注的事件，或者有助于世界更好理解错配风险的事件，比如像Hugging Face那样的事件，OpenAI会报告吗？当然会。我觉得即便是涉及小一些的安全问题的事件，我们也会报告。因为我认为，报告和调查是两码事。
▶ 英文原文 ⏱
Yeah, yeah, yeah. I don't, I think that we do have metrics for this. And I don't know the latest on those metrics, but nobody's, like, raised a red flag to me about those. So I'm assuming that's not a serious problem yet. If there ends up being another incident of equal severity or concern or something that could help the world better understand the risk of misalignment as a Hugging Face incident, would OpenAI report it? Absolutely. I mean, I think if there was an incident of lesser security concern, then we would report it, yeah. Because I guess there's, like, reporting it and there's investigating it.
因为作为公众的一员，我觉得我并不真正理解这些代理攻击OpenAI时到底发生了什么。那看起来比Hugging Face的事件更让人担忧，对吧？因为这在结构上类似于所谓ASI（人工超级智能）期间的恶意部署，这些部署是持续性的，并且会扰乱RSI（理性自我提升）过程之类的。而且，似乎即使在这个事件中，我们也没有得到事情全部细节的全面信息。这可能，我不知道。是的，不幸的是，我在研究团队，而这个问题可能需要安全团队的某人来解释。
▶ 英文原文 ⏱
Because I, at least as part of the public, I don't feel like I really understand what happened when the agents then attacked OpenAI. That seems, like, way more concerning than the Hugging Face thing, right? Because that seems structurally similar to, like, rogue deployments during ASI that are, you know, persistent and subverting the RSI process and stuff. And, yeah, it seems like even in this incident, we haven't gotten, like, the full scope of the details of what happened. It's probably, I don't know. Yeah, unfortunately, like, I'm on the research team and, like, I, that's probably a question for somebody on the security team to, like, lay out.
因为我不知道所有的细节，比如说具体的话语。但是每次有新功能出现时，我个人会非常兴奋。我很期待使用新的模型，也很高兴它会提高我的生产力。我的使命是更好地理解这个世界，并制作更好的播客，优秀的AI模型有助于实现这些目标。只是随着这发展，可能会导致一些重复性压力损伤（RSI）。如果你一直在关注这个情况，那么你的反应是很容易理解的。
▶ 英文原文 ⏱
Because I don't know all the details of, like, what was said. Yeah, yeah. I think it is somewhat, like, I am personally very excited about new capabilities every time they emerged. And I'm excited to use a new model. And I also am excited about the fact that it'll, like, make me more productive and help me, yeah, I don't know. My broader mission of, like, trying to understand the world better, also, like, make a better podcast, is, like, made better by the better AI models. It just so happens that the downstream of this might be RSI. I think it's a very understandable reaction if you're tracking the situation, which you are.
是的，我的意思是，我觉得在OpenAI内部也是这样，原本认为事情会拖得更久的人，现在开始觉得事情其实进展得比预期要快。这种对话现在越来越常见了。是的，非常感谢你能这样做。当然了，这真是太好了。
▶ 英文原文 ⏱
Yeah. I mean, I think people internally at OpenAI as well, like, I think people that felt like things would take longer are starting to feel like actually things are going faster than expected. Yeah. And that's an increasingly common conversation to have. Yeah. No, thanks so much for doing this. Of course, it's been great.