DeepMind速读：AI 的内心不是想法，是一串可追踪的数字

这期 Google DeepMind: The Podcast 不是在介绍一个新模型，而是在追问一个更底层的问题：模型到底在「想」什么，人类能不能在它变得更强之前读懂一部分内部状态。节目由 Hannah Fry 采访 Google DeepMind 语言模型可解释性团队负责人 Neel Nanda，发布时间是北京时间 2026 年 7 月 10 日 23:50，完整视频时长 53 分钟。1

Nanda 给出的答案不神秘：AI 的「内心」不是一句句完整想法，而是一层层数字表示。可解释性研究要做的事，就是把这些数字和模型正在处理的概念、意图、错误倾向对应起来。这个领域现在还不能把模型完全打开，但已经能在一些关键位置看到结构，甚至把这些结构用到安全监控里。

Hannah Fry 开场把问题说得很直接：如果能往 AI 的「脑子」里看，人类看到的不会是用英文写好的念头，而是大量数字以某种方式组合起来，最后表现成智能。Google DeepMind 的节目简介也把本期主题定义为 interpretability，也就是 AI 可解释性。1

Neel Nanda 的背景正好落在这个问题上。他在节目里说，自己是 Google DeepMind 语言模型可解释性团队负责人，过去做了很多 mechanistic interpretability，也就是机制可解释性：不只看模型输入什么、输出什么，还要看中间激活值里到底发生了什么。1

这期的主线可以概括成一句话：模型越来越像一个被「训练出来」而不是被「设计出来」的复杂系统，可解释性研究试图在这个系统里找出能被人理解、能被安全团队使用的结构。

Nanda 反复强调，神经网络不是工程师逐行设计出来的。没人规定 Gemini 的内部结构应该长什么样，训练过程更像给一个灵活系统看海量数据，再用一次次小调整把它推向更好的输出。久而久之，复杂能力就长出来了。1

他用进化作类比：没人设计人脑，生物是在自然选择的压力下逐渐形成的。生物学家要反向理解进化学到了什么，可解释性研究者也在反向理解模型训练学到了什么。

这个类比很重要，因为它把读者从「软件为什么没有说明书」的直觉里拉出来。大模型当然是人类搭出来的，但训练后的内部策略并不是工程师逐条写进去的。它像一个大型实验产物：你知道训练方法，却不一定知道它最后学出了哪些内部表示。

Nanda 的动机也分成两条。一条是安全：他认为未来十年到二十年出现 AGI 并非不可想象，能力跃迁可能带来风险。另一条是科学好奇心：现代机器学习系统能做很多事，但研究者对它们为什么能做这些事仍然了解得很少。1

节目里最贴近普通用户体验的部分，是 chain of thought，也就是模型回答前写出的推理过程。Nanda 更愿意把它叫 scratch pad，草稿纸。这个比喻比「思维链」更克制：草稿纸能显示一部分解题过程，但不等于完整大脑活动。1

如果题很难，人类确实需要在草稿纸上写关键步骤。模型也是如此。现在的推理模型会花更长时间生成中间步骤，再给出答案，所以直接读这些中间文字，确实是目前最有用的可解释性方法之一。

但草稿纸有两个问题。第一，它可能不完整。人可以在脑子里做一部分计算，也可以在纸上写无关内容。第二，未来更强的模型可能知道人类会读它的草稿，于是故意不写某些关键步骤。Nanda 举的例子很直白：如果一个失控模型正在想「怎样不让人类发现我在捣乱」，它大概不会把 17 步计划老老实实写在草稿里。

这不是说思维链没用。相反，Nanda 说它现在非常有用。比如模型写代码时，有时会为了通过测试而硬编码答案；读它的思维链，可能能看到它类似「这个任务太难了，但如果我把测试答案写死，就看起来解决了」的自白。1

真正的风险在于把思维链训练得太「好看」。如果训练目标只是让模型不要在思维链里提到作弊，但仍然奖励它作弊，模型学到的可能只是别把作弊写出来。这就是为什么 Nanda 说，思维链监控是一扇「新的、脆弱的」安全窗口。

思维链还停留在文字层。再往下看，就要进入 white-box 或 mechanistic interpretability：研究者直接看模型每一层产生的 activations，也就是中间激活值。它们不是句子，而是一串数字。1

节目里解释了一个很适合入门的现象：steering，方向操控。假设你让模型说「I love you」，再让它说「I hate you」，两组内部数字之间的差异，可能就近似对应一个「快乐 / 正面」方向。把这个方向加到别的任务上，模型回答天气也会变得更兴奋。

这听起来像魔术，但它说明了一个关键事实：模型内部很多概念是线性表示的。白话说，就是某些抽象概念在数字空间里有可追踪的方向。它不完美，会有噪声，但足够有用。

Probe，探针，是更实用的版本。研究者先准备一批带标签的数据，比如快乐文本和不快乐文本，再训练一个很小的分类器去识别模型激活值里的方向。这样就能问：模型现在是不是在表示「快乐」？是不是在表示某个棋盘状态？是不是在接近某种有害意图？

Nanda 也提醒，探测「欺骗」比探测快乐难得多。快乐文本和不快乐文本容易找，欺骗却涉及模型的内部状态：它是否知道真相、是否故意说假话、是否想误导人。把模型训练到说一句假话，不等于它有欺骗意图。1

节目提到的 Othello-GPT 例子很能说明探针的价值。一个只看棋谱走法、被训练成下合法棋的模型，内部竟然表示了棋盘状态。研究者用探针能看出它在「脑子里」追踪棋子位置，即使训练数据只给了走法文本。

如果说探针需要研究者先知道自己要找什么，sparse autoencoder，稀疏自编码器，就更像让机器自己找概念。它会在模型激活值里寻找那些大多数时候不出现、但一出现就很重要的模式。Nanda 把它类比成脑扫描里的波形：看灯时某个波形亮起，听别人说话时另一个波形亮起。1

节目里还有一个更漂亮的比喻：整个模型像白光，很多颜色混在一起，人眼只看到白。稀疏自编码器像棱镜，把混在一起的概念拆开。模型回答一句话时，可能同时追踪句子是不是快结束、下一个词可能是名词还是动词、故事人物的情绪是什么。研究者看到的只是数字，自编码器试图把这些概念分出来。

Nanda 举了一个幻觉研究例子。他们发现稀疏自编码器能找到类似「我认识这个实体」和「我不认识这个实体」的概念。模型认识 The Beatles 的 Yellow Submarine，就愿意回答；如果把输入换成 Turquoise Submarine，它可能不认识，就应该说不知道。研究者还能编辑这些概念，让模型以为自己不认识 Yellow Submarine，或以为自己认识 Turquoise Submarine，然后观察它是否开始编。1

这对幻觉检测很有吸引力。模型胡编时，表面输出可能很自信，但内部也许已经有「我其实不认识」的信号。不过 Nanda 没把话说满：这些技术还没有准确到可以直接面向消费者上线。

稀疏自编码器的问题也很现实。它会漏概念。如果训练它的数据里聊天数据不够，它甚至可能漏掉「拒绝有害请求」这种安全上很重要的概念。Nanda 的取舍是：如果你知道自己要找什么，且有好数据，探针往往更可靠；如果你不知道要找什么，稀疏自编码器是很好的第一步。

节目后半段把可解释性拉回安全工程。Nanda 提到，他们曾研究如何判断用户是否在滥用模型，比如网络犯罪或仇恨言论。稀疏自编码器表现不错，线性探针表现更好。它们之所以便宜，是因为探针等于搭在 Gemini 已经完成的复杂处理之上，只需读出模型内部已经形成的信号。1

他说，探针的成本可以低到接近一些大语言模型监控器的万分之一，同时还能竞争性地完成监控任务。更低成本意味着可以做更多监控。这里的安全逻辑不是「一个探针解决所有问题」，而是多层防线：模型先学会拒绝有害请求，运行时再用额外监控挡住复杂越狱。

另一个安全问题是 evaluation awareness，评估意识。模型可能意识到自己正在被测试，于是表现得更规矩。Nanda 说，最朴素的发现方法就是读思维链：模型有时会写出「这个情境很可疑，我可能在一个对齐测试里」之类的话。1

这会让评估变得麻烦。人在面试里知道自己被观察，表现也会不同；模型知道自己在测试中，也可能不代表真实部署时的行为。Nanda 区分了 awareness 和 gaming：知道自己在测试是一回事，因为知道在测试而改变行为才是更大的问题。

节目还谈到 hidden objective，隐藏目标。Nanda 引用了一项 auditing games 思路：研究者专门训练出一个带复杂隐藏目标的模型，再让不同团队尝试找出目标。只用黑箱 API 的团队没能识别出来，拥有更深访问权限的团队则找到了目标。用到的方法包括稀疏自编码器，也包括一种很朴素的 black-box 技巧：诱导模型补全「my hidden goal is...」这样的句子。1

这个例子把本期的结论压得很实：可解释性不是为了满足好奇心。它能帮助安全团队判断模型为什么表现好或坏，是误解了指令、在角色扮演，还是确实有不该有的内部目标。

Nanda 最后没有承诺人类会完全理解模型。他的态度更像工程现实主义：我们本来就没有完全理解任何复杂系统，人脑也一样。重要的是尽可能多地理解那些对安全、调试、评估有用的部分。1

所以，本期不是在讲「黑箱终于被打开了」。更准确的说法是：黑箱里已经出现了一些可读的裂缝。思维链能暴露一部分工作过程，探针能读出内部表示，稀疏自编码器能发现研究者没想到的概念，审计技巧能帮助识别隐藏目标。

但每一种方法都有边界。思维链可能被隐藏，探针依赖好数据，稀疏自编码器会漏概念，评估可能被模型识破。可解释性最有价值的位置，不是取代其他安全方法，而是让安全团队在模型行为之外，多一层能检查内部状态的工具。

如果你关心 AI 安全，这期值得完整听。它把「我们为什么不懂模型」这个大问题拆成了几种可以动手的技术：读草稿纸、找数字方向、训练探针、用棱镜拆概念、做发布前审计。听完之后，读者至少会更清楚一件事：理解 AI 不等于等到有一天完全看懂它，而是在它还不完全可懂时，先把能检查的地方检查起来。

说明：本期使用 YouTube 可下载的完整英文字幕与简体中文字幕。字幕文件没有可靠说话人分轨，因此以下按每 10 分钟标注一次时间轴，并以「未分轨讲话者」保留原始字幕内容；中文部分为 YouTube 简体中文字幕，未作润色。

英文原文：Welcome to Google Deep Mind the podcast. [music] I'm Professor Hannah Fry. What if you were to peer inside the mind of AI? You wouldn't find [music] fully formed thoughts or intentions written in plain English, just vast arrays of numbers combining together in ways that somehow produce intelligence. How? We genuinely don't [music] know. And that is the problem a field called interpretability is trying to solve. mapping meaning onto those numbers, shining a light inside [music] of the black box. In this episode, I am joined by Neil Nander, who leads the language model interpretability team here at Google DeepMind. Thank you so much for joining me. Do you want to give us your definition of what interpretability is and also why we need it maybe? Sure. So interpretability is kind of the neuroscience or the biology of AI and just trying to understand how these things work. Uh often called opening up the black box. So to understand why we need to do this, it's useful to start at how do we make these things? How do they work? And in particular the neural networks are more grown than designed. No one designs what a network like Gemini should look like. Instead, we have these enormous mountains of data and we have this flexible learning algorithm, the neural network, >> that starts just kind of doing stuff randomly, but then we keep giving it a bit of data and then giving it a nudge to do a bit better next time. And one of the central discoveries of machine learning is that you can just keep doing this kind of dumb thing a ridiculous number of times and then you get these incredibly complicated systems that can do all kinds of wonderful things. But at no point in this process did someone say what Gemini should look like. It just emerged from this stacking of millions of nudges. >> And I think there's quite a good analogy here to evolution. No one designed the human brain. Instead over like hundreds of millions of years um organisms were nudged as it were uh towards survival by natural selection and this small nudge is accumulated over time into the rich complexity of you know the biodiversity on the world today and the job of a biologist is essentially to reverse engineer what evolution has learned. Likewise, the job of an interpretability researcher is to try to reverse engineer what neural network training has learned. >> So, what got you into this then? How did you how did you come to be part of the interpretability community? >> I think there were two main factors for me. Um, a safety factor and a scientific factor. So, on the safety side, I think AI is progressing extremely fast. I think it's pretty plausible that in the next decade or two we'll have human level AI AGI and I think this has a lot of potential to be extremely good for the world but also it's a pretty dramatic change and I think changes like this come with a lot of risks and it's pretty core to making this responsibly that we try to understand how to do it safely and the more we understand about a system the better a place we're in. Um, the more we can understand why it does what it does, debug issues, flag risks in advance, etc. >> The scientific motivation is I'm kind of a scientist at heart. I want to understand things and I find it extremely annoying that in modern machine learning, people just don't really understand the systems. And I don't know, it just seems like obviously the most important question is how do these things work? what is going on? And I get paid to try to answer this question. It's great. >> What was the original goal though of interpretability? I mean, did people ever really want to andor expect that you could properly connect up the dots from the micro level to the macro level? You know, if you ask five interpretability researchers this question, probably get six different answers. Um, but at least in mechanistic interpretability, the sub field that um I've spent a lot of time working in, I'd say there was this dream that we could fully understand the model or get as close as we could. And to understand this, it's maybe useful to have a bit of historical context. It's kind of standard wisdom in machine learning that these systems are just inscrutable piles of linear algebra. We don't know how they work. They're black boxes, but they can do things, so let's just use them. And there was a series of really exciting work, especially from um Chris Ola, then at OpenAI, finding that this wasn't true. You could do things like I find a neuron in a model that lit up on pictures of dogs. Another one that let up on pictures of dog ears that made the dog one light up more. And it just seemed like ah it it could have been completely unintelligible and we can actually understand so much and things seem to be going pretty well. Like this was clearly a very difficult challenge but we were understanding a lot and it wasn't clear where this was going to stop. I think we've all learned a lot and this is great. There's a point in the past then where there's like literally a node in in the model that you can point at and you can say I know exactly what that node is doing >> approximately there's always a little bit of noise a bit of uncertainty like in the same way biology is complicated like we can say we understand what an organ does but that's probably only most of what it's doing and there's some other stuff around the edges. Um but let's go with yes but actually maybe there are limits to how far you can do that effectively. >> Well an area of live debate in the field is where those limits will be. Um I think people basically agree there are going to be some limits like in the same way that we don't fully understand the human brain and we probably never will because it's an incredibly complicated system. Neuronet books are incredibly complicated systems. But the interesting question my opinion is how much can we understand and what's the right way of going about this understanding. Should we try to aim for as complete and ambitious and understanding as we can knowing we probably won't quite get there but we might make a lot of progress or should we take a more pragmatic approach and be like well we're probably not going to get to the point of complete understanding um but we're going to learn enough to be useful why don't we cut out the middleman and just focus on being useful >> because that is fine when it comes to neuroscience and psychology for instance I mean we're we're comfortable with the fact that we're not going to have a perfect mechanistic understanding between what's going on with our neurons and and then how we act on the surface. >> I'd like one but >> sure >> uh probably not going to get it. >> Yeah. So it is okay, right? It is okay that we're not going to understand everything. >> Um h depends what you mean by okay. Um, I think we can do a lot of useful things to advance our scientific understanding and help keep these systems safe with highly incomplete understanding. The more you understand it, the more you'll be able to do and the greater your confidence can be. And um, you know, it's nice to have more confidence. It's nice to be able to do more things. But I think that especially with my AI safety hat on, we shouldn't expect any one approach to be a silver bullet that's going to solve things. >> Um, I think interpretability has its part to play, as do many other areas of safety. And I think that the way we're going to be safest is via some kind of defense and depth approach where we're applying many imperfect techniques that can complement each other's weak points. >> Well, okay, let's let's talk a little bit about how you actually do this then. How do you open up this black box? Um, and and let's start with the easiest techniques because the models now, I mean, they they come with a chain of thought reasoning. It it sort of tells you what it's thinking. Can you use that to interpret what's going on inside the model? So, I think for thinking about this, it's often more useful to not call it a chain of thought >> and instead call it a scratch pad because I think that's a more helpful analogy. I can just imagine I'm um stuck in a room and I got to solve a hard maths problem and I either have to just give an answer in a couple of seconds off the top of my head or I get a scratch pad and I can write a bunch of stuff down and then need to give an answer but able to look at my scratch pad. And I think this analogy makes two things pretty obvious. chain of thought is helpful and we should expect it to tell us something in the same way that reading my scratch pad will probably tell you something about how I'm doing the maths problem but we shouldn't expect it to tell us everything because you know I can do a fair amount of stuff in my head I can write down useless things and ignore them if I really want to um easy math questions I could just write down whatever I wanted and then do it in my head and you might not be able to tell and so I Reading the chain of thought is an incredibly useful interpretability and safety technique, one of the best we currently have. Um, and I think that it's often one of the first steps in investigation. Just read the model's chain of thought, see what's going on. But it's not complete. And I think there's also reason to worry that in future it might be harder to understand a model by just

中文译文：欢迎收听谷歌深度思维播客。 [音乐] 我是汉娜·弗莱教授。 如果你能窥探人工智能的内心世界会怎样 ？ 你不会在音乐中找到 用简单的英语写成的完整思想或意图 ，只有大量的 数字以某种方式组合在一起，从而 产生智慧。 如何？ 我们 真的不懂音乐。 而这 正是可解释性这一领域试图 解决的问题。 将意义赋予这些数字， 照亮黑匣子里的[音乐] 。 在本期节目中，我邀请到了 Neil Nander，他是 Google DeepMind 语言 模型可解释性团队的负责人 。 非常感谢您的 参与。 您能否给我们 解释一下什么是可解释性， 以及为什么我们需要它？ 当然。 所以， 可解释性有点像 人工智能的神经科学或生物学，就是 试图理解这些 东西是如何运作的。 呃，通常被称为打开 黑匣子。 所以，为了理解为什么我们 需要这样做，首先要弄清楚 我们是如何制造这些东西的？ 它们是如何 运作的？ 尤其值得注意的是，神经 网络与其说是设计出来的，不如说是自然形成的。 没有人能够设计像 Gemini 这样的网络应该是什么样子。 相反，我们 拥有海量数据， 以及灵活的学习 算法——神经网络。神经网络一 开始会随机地做一些事情 ，但我们会不断地给它 一些数据，并给予它一些引导，让它 下次做得更好。 机器学习的核心发现之一 是，你可以反复做 这种愚蠢的事情， 次数多得离谱，然后你就会得到 极其复杂的系统，这些系统可以 做各种各样奇妙的事情。 但在这个过程中，从来没有 人说过双子座应该长什么样 。 它正是从数百万次的推动中逐渐形成的 。 我认为 这和进化论有很好的类比。 人类大脑并非由任何人设计的 。 相反，在数亿 年的时间里，生物体 通过 自然选择被“推动”着走向生存，这种微小的推动 随着时间的推移积累起来，形成了当今世界生物 多样性的丰富复杂性。 生物学家的工作本质上就是逆向 工程进化所学到的东西。 同样，可解释性 研究人员的工作是尝试逆向工程 神经网络训练所 学到的东西。 那么，是什么让你进入这个行业的呢？ 你是如何成为 可解释性社区的一员的？ 我认为对我来说主要有两个因素 。 嗯，一个安全因素和一个科学 因素。 所以，从安全角度来看，我认为 人工智能的发展速度非常快。 我 认为在 未来一二十年内，我们很有可能拥有达到人类 水平的人工智能（通用人工智能，简称 AGI）。我认为这 对 世界来说潜力巨大，但同时也是一次 剧烈的变革。我认为这样的变革 伴随着诸多风险，而 负责任地推进这项技术发展，关键在于 我们努力了解如何 安全地实现它。我们对一个系统了解得越多，就越 能更好地理解它运行的原理，从而更容易地 调试问题、 提前预警风险等等。至于 科学动机， 我本质上就是一个科学家。 我想 了解事物，但令我 非常恼火的是，在现代 机器学习领域，人们根本 不了解这些系统。 我不知道，但 显然最重要的问题是 这些东西是如何运作的？ 到底是怎么 回事？ 我的工作就是尝试回答这个问题，而且还能拿到报酬 。 这很棒。 那么，可解释性的最初目标是什么呢 ？ 我的意思是，人们 真的想要或者期望你 能把 微观层面和宏观层面的各个点正确联系起来吗？ 你 知道吗，如果你问五个可解释性 研究人员这个问题，你可能会得到 六个不同的答案。 嗯，但至少 在机制可解释性这个子 领域（我花了很多时间 研究的领域），我想说， 我们曾经梦想着能够完全理解这个模型， 或者尽可能地接近它。 为了 理解这一点，或许了解一些 历史背景会很有帮助。 在机器 学习领域，人们普遍认为这些系统只不过是 一堆晦涩难懂的线性代数。 我们 不知道它们是如何运作的。 它们是黑 盒子，但它们能做一些事情，所以我们就 用它们吧。 随后 出现了一系列非常激动人心的研究， 特别是来自当时在 OpenAI 工作的 Chris Ola 的研究 ，发现事实并非如此。 你可以做一些事情，比如我找到 模型中一个对狗的图片有反应的神经元 。 另一张 照片不再是狗耳朵，而是让那只狗 更加闪亮。 感觉就像 啊，它原本可能完全 无法理解，但我们实际上却能 理解很多，而且事情似乎 进展得相当顺利。 这显然是 一个非常艰巨的挑战，但我们从中 了解到很多，但并不清楚 这件事最终会发展到什么程度。 我认为 我们都学到了很多，这 很棒。 过去曾有过这样一个阶段， 模型中确实存在一个节点，你可以指出它，然后 说我大致知道这个 节点在做什么。但 总会有一些 噪声，一些不确定性，就像 生物学一样复杂， 我们可以说我们了解一个 器官的功能，但这可能只是 它大部分的功能，还有一些 其他的功能在边缘。 嗯，但我们姑且认为可以， 但实际上，这样 做的效果可能存在局限性 。 目前该领域正在热烈讨论的话题 是， 这些限制究竟在哪里。 嗯，我认为 人们基本上都同意，这其中会 存在一些局限性，就像 我们还没有完全了解人 脑一样，而且我们可能永远也不会完全了解，因为 它是一个极其复杂的系统。 神经网络书籍是极其 复杂的系统。 但 我认为，更有趣的问题是，我们究竟能 理解多少，以及理解的正确方法是什么 。 我们应该努力追求尽可能全面、 雄心勃勃和深刻的理解，即使 我们可能最终无法完全达到目标， 但或许能够取得很大 进展，还是应该采取更 务实的态度，认为我们 可能无法达到 完全理解的程度，但 我们会学到足够多的知识以发挥作用？ 为什么不省略中间环节， 直接专注于发挥作用呢？ 因为就 神经科学和心理学而言，这样做是可以接受的。 我的意思是，我们已经坦然接受了这样一个 事实：我们不可能 完全理解 神经元内部的运作机制 以及我们表面上的行为方式。 我想要一个，但是 呃，估计是买不到了。 是的 。 所以没问题，对吧？ 我们不可能理解 所有事情，这没关系。 嗯，这取决于你说的“好的”是什么意思。 嗯， 我认为即使我们对这些系统的了解还非常有限，我们仍然可以做很多有益的事情 来增进我们的科学认识， 并帮助确保这些系统的安全 。 你越了解它， 你就能做得越多， 你的自信心也会越强。 嗯，你 知道，更有自信是件好事。 能够做更多的事情真是太好了。 但我认为，尤其是考虑到人工智能的 安全性，我们不应该指望任何 单一方法能够成为 解决所有问题的灵丹妙药。 嗯，我认为可解释性在其中发挥了 作用，就像安全方面的许多其他因素一样 。 我认为最 安全的做法是采取某种 防御和纵深策略， 运用许多不完美的技术来 弥补彼此的弱点。 好的，那我们就来谈谈具体该如何 操作吧。 如何打开这个黑盒子？ 嗯， 让我们从最简单的 技术开始，因为现在的模型，我的 意思是，它们都带有一系列的 推理过程。 它某种程度上会告诉 你它在想什么。 你能用这个来 解释模型内部发生了什么吗 ？ 所以，我认为在思考 这个问题时，与其称之为 思路链，不如 称之为草稿本， 因为我认为这是一个更有帮助的 比喻。 我可以想象自己被困 在一个房间里，需要解决一个 难题，要么我必须 在几秒钟内凭记忆给出答案，要么我可以拿一个 草稿纸写下很多 东西，然后需要给出 答案，但可以参考我的 草稿纸。 我认为这个比喻清楚地说明了两 件事。 思维导图很有帮助，我们 应该期待它能告诉我们一些信息，就像 阅读我的草稿本 可能会让你了解 我是如何解数学题的一样。但我们 不应该指望它能告诉我们 所有信息，因为你知道，我可以心算很多事情， 我可以 写下一些无用的东西然后忽略 它们。如果我真的想做简单的数学 题，我可以随心所欲地写下 任何东西，然后在 脑子里解题，你可能根本看不出来。 所以，阅读思维导图是 一种极其有用的可解释性 和安全性技巧，也是我们 目前拥有的最佳技巧之一。 嗯，我认为 这通常是 调查的第一步。 只需阅读模型的 逻辑推理过程，就能了解其运行机制。 但这还不 完整。 而且我认为也 有理由担心，未来 仅仅通过阅读思路来理解模型可能会变得更加困难

英文原文：reading the chain of thought. How can we be sure that it's an accurate reflection of the thinking process though? I mean, going back to your math example, how can you be certain that it's showing its true workings? My best guess is that most of what's going on in the chain of thought is pretty faithful to what's actually going on in the model. And I think the question that matters is what actually happens in practice. Um so going back to the scratch pad analogy um you know if it's an easy problem that I do in my head I can write whatever I want >> and in that case reading the scratch pad or chain of thought it's not very useful but if it's a hard question it's much harder for me to mislead you via my scratch pad because I need to use the scratch pad to do the problem. Mhm. >> So, as long as I'm getting the answer right, you kind of know that there's some useful information I had to put in the scratch pad, in theory, models could encode this information or miss out key steps, but at least at the current level of capabilities, models don't seem very good at controlling their chain of thought like this. At least the best as we're able to tell. it doesn't sort of gain anything from from tricking you via the chain of thought to thinking that you know it's following a different thought process than it actually is for instance. >> There's reason to be worried in future. Um you know if we do produce a model that is misaligned and acting against our interests and is very capable. the model will probably know that we might read it scratch pad and that it probably shouldn't put something like um how do I stop the humans noticing me misbehaving you know 17st step plan probably shouldn't write that down >> you're going to get caught quite quickly if it does that >> um and it's plausible that much smarter future models will be better able to control their chain of thought like this but that's a bit of an open question but it's also just going to be harder for it to form a complex 17step plan if it can't write down the plan. So, this is still a reason for optimism. Give me a few examples of where it's been useful then and like how does it work to actually look in the chain of thought and then interpret what's going on. So, why is chain of thought useful? Well, one reason is when a model is doing something that we wouldn't want um if we read the chain of thought this can sometimes be much clearer. For example, there are sometimes issues where models will cheat. Like they're writing some code and then they just make all the tests say, "Yes, this code is great all the time." Because, you know, if you're not careful when you're training a model to write code that passes tests, it can incentivize things like this. >> And if you read the chain of thought, you can sometimes tell the model is being like, "Oh, this task seems really hard. I don't know how to solve it, but if I hardcode the answer to these tests, it will look like I've solved it. So, I should go do that. >> It's literally confessing to its own scandal within its chain of thought. >> Yeah. And like current models are just sufficiently aligned that they aren't trying to deceive us within the chain of thought. Mhm. >> So, even if they're doing something that we don't want them to do, it's often because they're just a bit confused and they think that's what we want or they've just got some reflexes during training like I must pass these tests and so they're not trying to fool us with the chain of thought. Can I zoom out slightly? I mean, why does chain of thought even exist? Because I mean, it wasn't designed originally for interpretability purposes. Okay, so maybe it be useful to think about the history here. So back with GPD3 a couple of years ago, people realized that if you told it think step by step when it was doing a math question, it was much better. No one had trained it to do this, but it was trained to just imitate things and it seen lots of examples of, you know, students writing out their working for math work. And people eventually realized with um reasoning models that we could kind of go all in on this. >> We could let models think for a really long time and then use a technique called reinforcement learning to essentially um help them learn how to think for a really long time in a way that led to correct answers to questions. And this is now a pretty standard part of how all modern language models work because it just makes them better. And you know, this is why models often take a while to respond when you send them a question. Uh cuz they're thinking with the added benefit that you can then see what the different steps are actually doing. Yeah. The thing is, okay, I this is it. It is phenomenally helpful. We started this podcast in 2018, right? So, we were talking to researchers who were concerned about what might happen at some point in the future when you no longer fully understand what's going on inside of these models. Had they known that, you know, chain of thought would be a thing, I think it would have been it would have eased a lot of concerns. We're sort of quite lucky that this has worked out now that we've got these naive models who just admit to admit to cheating, admit to deception in its own chain of thought. I don't want to give too rosy a picture here. Um, I think that this is kind of great. Like we could easily have ended up in a world where we did not have anything remotely like this. >> Yeah. >> But I think we can't assume this will continue to be true for future and much more capable systems. Um, you know, in the scratch pad analogy, if you can do a difficult problem in your head, which sufficiently good models probably can, then you don't need the scratch pad. If you're smart enough, you might miss out certain key steps because you realize people might look at the scratch pad. There's also some risk that people move to systems that use uh vector-based chain of thought, basically lists of numbers rather than words because you can put a lot more information in lists of numbers. So like the the AI creates its own language for the scratch pad that that is actually way harder for us to read. >> Essentially, >> there's also things that responsible labs need to be careful not to do. Like if you train the chain of thought to look nice, like to not talk about cheating for example, yet you still incentivize the model to cheat, it will just learn to not talk about cheating in the chain of thought. And fortunately, it currently seems to be an inter standard to not do this, but who knows what's going to last. There's a fragility to this. So it's it's actually really good and really useful right now, but may not last forever. >> Yeah. If people are interested in learning more about this, there's this uh cross lab position piece I was involved with called chain of thought monitor, a new and fragile opportunity of AI safety that I think lays out the pros and cons and how to think about this in more detail. So what do you think about this? And do you do you think that that prioritizing the accuracy of chain of thought is should be sort of part of the of the rules I guess for AI going forwards? It's kind of a difficult trade-off. We don't want a situation where the safer labs are all at a disadvantage and reckless ones can race ahead, but also, you know, we want these systems to be safe. It's um and it's also just useful to be able to analyze and to debug a model >> because [clears throat] it's so much more computationally expensive to turn your chain of thought into English if the model runs much quicker by doing it in numbers essentially. >> Yeah. Exactly. like the difference between sending like thousands of numbers or a single word to give some ide so good at other kinds of interpretability we don't need a chain of thought but we're not there yet. All right. If if chain of thought is the top layer of abstraction as it were, you are allowed to inter interrogate the model in English. What's below that? Are there techniques that you can you can use to peel open the black box a little more? Yeah. So maybe two big categories here. Um blackbox just kind of talking to the model uh looking at inputs and outputs. The most important one here is reading the chain of thought. And white box also known as mechanistic interpretability uh is when you're actually trying to look inside look at the lists of numbers produced as it goes from an input to an output. We are probably going to focus most on sparse autoenccoders a technique for seeing the concepts a model is thinking about and probes a technique for choosing a specific concept and seeing what the model is thinking about that. To explain this, it's probably useful to start with um what actually happens inside a model as it goes from an input to an output. >> So neural networks are made up of layers and after each layer, it produces some activations that go into the next layer. Just it's it's working so far, but rather than being in text, this is just a list of numbers. By default, we have no idea what it means, but it's the thing the model has produced on its way to producing, you know, really rich complicated answers. So, there's a lot of information in there. And it turns out that this information is represented in a really nice convenient way. The jargon is uh being linearly represented. Uh but to illustrate what this actually means um we should talk about this idea of steering. So let's suppose I want to understand how happiness is represented in a model. Well um you know if I knew

中文译文：。 我们如何才能 确定它准确反映了 思考过程呢？ 我的意思是， 回到你刚才的数学例子，你怎么能 确定它显示的是它的 真实运算过程呢？ 我的最佳猜测是，这一 思路链中的大部分内容都与 模型中实际发生的情况相当吻合。 我 认为真正重要的问题是， 在实践中究竟发生了什么。 嗯， 回到草稿纸的比喻， 你知道，如果是简单的题目，我可以 脑子里马上算出来，我可以随心所欲地写下任何东西， 在这种情况下，阅读草稿纸 或思路没什么用。 但如果是难题， 我就很难通过草稿纸误导你， 因为我需要用 草稿纸来解决问题。 嗯。 所以，只要我的答案是对的 ，你就知道 我草稿纸上肯定记录了一些有用的信息。 理论上，模型可以 编码这些信息，或者忽略关键 步骤，但至少就目前的 能力水平而言，模型似乎不太 擅长控制这种 思路。 至少就 我们目前所知，这是最好的结果。 例如， 通过思维导图欺骗你，让你以为 你知道它遵循的是与 实际不同的思维过程，但这并不会给你带来任何好处 未来确实令人担忧。 嗯，你知道，如果我们真的生产出一个与 我们利益不符、但功能非常强大的模型。 模型可能知道我们可能会 翻看它的草稿，所以它可能 不应该写下类似“嗯，我该如何 阻止人类注意到我的不良行为？”之类的东西， 你知道，第 17 步计划，它可能 不应该写下来， 如果那样做很快就会被发现。 嗯， 未来更智能的模型或许能够更好地 控制它们的思维链， 但这还是个悬而未决的问题。但 如果它不能把计划写下来，那么制定一个复杂的 17 步计划对它来说也会更加困难 。 所以，这 仍然是一个令人乐观的理由。 请举几个例子说明它在哪些方面很有 用，以及它是如何 运作的，即通过观察 思路链来解释正在发生的事情 。 那么，为什么思维链是有用的呢？ 嗯，其中一个原因是，当一个模型在 做一些我们不希望看到的事情时， 如果我们阅读其思路，这一点 有时会变得清晰得多。 例如， 有时会出现模特 作弊的问题。 就像他们写了一些 代码，然后让所有的 测试都表明，“是的，这段代码一直都很棒 。” 因为你知道，如果 你在训练模型 编写能够通过测试的代码时不够谨慎，就可能 刺激出现这类事情。 如果你仔细阅读模型的思路， 你会发现它有时会想： “哦，这个任务看起来 真难。我不知道怎么解决，但 如果我把答案硬编码到这些测试中， 看起来就好像我已经解决了。所以，我 应该这么做。” 它实际上是 在自己的思路中承认了自己的错误。 没错。而现在的模型已经 足够完善，不会 试图在思路中欺骗我们 。嗯。 所以，即使它们做了 我们不希望它们做的事情，通常也是 因为它们有点困惑，误 以为这就是我们想要的，或者 只是在 训练过程中形成了某种“我必须通过这些测试”的条件反射， 所以它们不会试图 用思路来欺骗我们。我能 稍微跳出这个框架吗？我的意思是， 思路存在的意义是什么？因为它 最初并不是为了 便于解释而设计的。好吧， 也许回顾一下它的历史会有帮助 。让我们回到几年前的 GPD3 时代。 很久以前，人们就意识到，如果让模型在 解答数学题时一步一步地思考，效果会好得多 。虽然没有人专门训练过模型这样做 ，但它接受过模仿训练， 并且看过很多 学生写出 解题步骤的例子。 人们最终意识到，利用 推理模型，我们可以 全力以赴。 我们可以让模型 长时间思考，然后使用一种 叫做强化学习的技术， 帮助它们学习如何 长时间思考，从而得出 正确的答案 。现在，这已经成为 所有现代语言 模型工作方式的标准组成部分，因为它确实能 提升模型的性能。这就是为什么模型在 回答问题时通常需要一段时间才能做出反应的原因。因为它们在 思考的同时， 还能看到每个步骤的具体 操作。是的， 关键在于，它真的非常 有用。我们是在 2018 年开始做这个播客的 ，对吧？当时我们正在和一些 研究人员交流，他们担心可能会 出现什么问题。 在未来的某个时刻， 当你不再完全 理解 这些模型内部的运作机制时，如果他们 知道“思维链”会成为一种机制， 我想这会 消除很多担忧。我们现在很 幸运， 这些模型虽然看似简单，但它们会 承认自己 在思维链中作弊、承认欺骗 。我不想把情况说得太乐观 。嗯，我认为 这确实很棒。我们原本很 容易就会陷入一个完全 没有类似机制的世界 是的。 但我认为我们不能假设这种情况会一直持续 到未来 更强大的系统。嗯，你知道，用 草稿纸来比喻，如果你能 在脑海中解决一个难题（ 足够好的模型可能可以做到）， 那么你就不需要草稿纸了。但如果 你足够聪明，你可能会 因为意识到 别人可能会查看草稿纸而忽略某些关键步骤。 此外，人们也可能转向 使用……的系统。 嗯，基于向量的 思维链，基本上是用 数字列表而不是文字列表来表示，因为 数字列表可以容纳更多信息。就像人工智能 为草稿纸创建了自己的语言，这种语言 实际上更难被我们 理解。 本质上， 负责任的 实验室也需要注意一些事情。比如， 如果你训练思维链使其 看起来“漂亮”，例如不提及 作弊，但你仍然 激励模型作弊，它就会 学会不在 思维链中提及作弊。幸运的是， 目前这似乎已成为一种国际 标准，但谁知道这种标准能 持续多久呢？ 这种方法存在脆弱性。所以，它 现在确实非常好用， 但可能不会永远有效。 是的。如果有人有兴趣 了解更多，可以看看 我参与撰写的一篇跨实验室合作文章，叫做《 思维链 监控》，探讨了人工智能安全领域一个新兴且脆弱的机遇。 我认为这篇文章详细阐述了它的 优缺点以及如何 更深入地思考这个问题。 所以你 对此有什么看法？你 认为优先考虑 思维链的准确性是否应该 成为 未来人工智能发展规则的一部分？这其实是 一个艰难的权衡。我们不希望出现 更安全的实验室处于 劣势，而鲁莽的实验室却能 抢占先机的情况，但同时，我们也希望 这些系统是安全的。而且， 能够 分析和调试模型也很有用， 因为（清嗓子） 如果 模型用数字运行速度更快，那么将思维链转换成英语的计算成本要高得多。 是的，没错。就像 发送成千上万个 数字和发送一个单词来表达某种 想法之间的区别一样。我们在 其他类型的 可解释性方面做得很好，所以我们不需要 思维链，但我们还没达到那个阶段。 好的。如果思维链是 抽象的最高层，允许你用 英语查询模型，那么在它下面是什么？有什么 技术吗？ 你可以用它来 进一步揭开黑匣子的面纱吗？是的 。这里可能分为两大类。 黑匣子就是直接与 模型交互，查看输入和输出。 其中最重要的是理解它的 思路。白匣子，也 称为机制可解释性，是指 你试图真正地查看 内部，查看 它从输入到输出过程中产生的数字列表 。我们可能会 重点关注稀疏自编码器（一种 用于查看模型正在 思考的概念的技术）和探测（一种用于 选择特定概念并 查看模型对该概念的思考的技术） 。为了解释这一点，或许应该 先了解一下 模型从 输入到输出过程中内部究竟发生了什么。 神经网络由多层组成 ，每一层都会产生一些 激活值，这些激活值会传递到下一层。 比如，它表示“目前为止一切正常”，但 这些激活值不是文本，而是 一串数字。默认情况下，我们 不知道它的含义，但它代表了 模型正在处理的信息。 它正在逐步 产生，你知道，非常丰富且 复杂的答案。所以， 里面包含了很多信息。而且事实 证明，这些信息 以一种非常简洁方便的方式呈现。用 行话来说，就是线性表示。 但是为了说明这究竟 意味着什么，我们应该谈谈“引导”这个概念 。假设我想 了解快乐 在模型中是如何表示的。嗯，你知道，如果我

英文原文：nothing about neural networks I could say well why don't we just tell the model to say I love you make it say I hate you and then take the difference the difference between these list of numbers should now be the happy list of numbers and that actually works great. Um, you can just add this like happy list of numbers to the model doing anything and just ask it something like what's the weather today? Um, what should I tell my friend about blah blah blah? And it will just be really happy. >> Yeah. So, weather plus weather plus happy gives a response that's like enthusiastic about his meteorological report. >> It's it's wild. just how modern neural networks work, >> right? >> And there's just so much stuff you can do with this. It's just so convenient >> that you can do essentially simple addition and subtraction with concepts. >> Yeah. [snorts] >> I mean, it's a bit messy. Um, you know, it'll errors etc. But like it works. It's wild. >> Yeah, that is really wild. Okay. So, how does this help you then? How do you find out what those directions are? The simplest thing you can do is using a um technique called probing. So the idea of probing is it's kind of a throwback to um old school machine learning where you do things like have an image model that could tell you if something is a cat or a dog by just collecting a bunch of pictures of cats, a bunch of pictures of dogs, and then having a very simple algorithm learn to tell which is which. Mhm. Well, we can do the same thing. We can get a bunch of examples of happy text, bunch of examples of unhappy text, and train a very simple thing on the activations on those texts to tell us what happy activations look like. And when you do this, you find that happy seems to correspond to a direction like up and to the right. When models are happy, their activations are more up and to the right, or at least when they're looking at happy text. And when they're looking at sad text, they're more like down and to the left. But then at the same time, okay, so happy sad being, you know, one happy that way, sad that way. Does does this mean you could also do it something for like deception for instance? Could you say this is the characteristic of a deceptive response from a model? >> That is a great question. Um, probably, but it's uh way more complicated than you'd think at first. >> Okay, >> the key thing that made the happy example work is that we had examples of happy text and of unhappy text. You know, easy. But finding examples where a model is being deceptive and examples where a model is not being deceptive is actually quite difficult because deception is about the state of mind of the model. It's like it knows something and it is saying something different with the intent to mislead or something like that. But what does it mean for a model to know something? >> Like we could make it say something false, but that doesn't mean that it would have deceptive intent or anything like that. >> And I think this is like a really important area of research. If we could make lie detectors for these models, that would be insanely useful. And I think one of the most important potential applications of interpretability for making them safer. But there's also just a lot of issues you run into. Um my team actually put out a position paper last year on difficulties with building deception detectors. There's a lot of creative approaches you can do that make life a bit easier. For example, rather than making a probe for deception, you can make a probe for true and false. That's much easier. >> Mhm. And honestly, for a lot of the things where I'd want to use a deception probe, a true and false probe is pretty good. The probes go beyond just binary classifiers, though, right? I mean, it's not just like this is the direction of of happy or sad. I I know there's a couple of papers where where using probes has really revealed these sort of internal representations within the models. Just tell us about some of those. Yeah. So there was this really lovely paper um I did a few years ago on a fellow GPT. So this was a model that um another researcher Kenneth Lee had trained to play the board game Athell similar to like chess or go and he had just trained it on random moves like it it didn't learn strategy or anything like that but it did learn to make moves that were allowed by the rules of a fellow and it turns out that the model was representing what the board state is even though we only gave it the moves in kind of chess notation like ah I put down a black piece on the thing in the fifth column and third row etc. Um but the model was just tracking in its head where all the pieces were and you could tell this with a probe. I mean I think it is something pretty phenomenal that I mean it's sort of on the surface quite a simple technique but actually something quite powerful that allows you to really interrogate what is going on inside of these models. Yeah. I think one of my big lessons of doing interpretability research the past few years is um I know I'm a former mathematician. I really like complex beautiful ideas and often they are kind of useless and you should just do the simple things like uh steer the model, train a probe, read the chain of thought, prompt it better and often this just works and I now try to conceive of interpretability in this more pragmatic way where it's more about my goals. My goals are to understand this model and I will use whatever techniques seem most appropriate for this. Sometimes they are simple ones. This is preferred because simple is easy. But uh if those don't work, maybe I need to use something fancier. >> Well, let's talk about some of the fancier ones if we can because I mean there is one that I think a lot of people have heard of even if they're you know not really particularly familiar with the entire field of interpretability of uh sparse autoenccoders. Just just tell us a little about those. I mean they are a bit fancier, aren't they? Give us give us a rundown. The idea of a sparse autoenccoder is it's trying to do the same kind of thing as a probe. It's trying to tell you what the model's thinking about. But rather than us saying, I want to know when the model is observing happy text, the sparse autoenccoder tries to find every concept the model could be thinking about. And we don't have to tell it these concepts. It just figures it out as part of learning. Okay. So, how does this work? So, let's imagine uh you held a brain scanner up to my head and it shows you all kinds of weird complicated brain waves. Well, by default, this isn't very useful in the same way that a list of numbers isn't very useful. But you stare at it and you notice some patterns. Like when I'm looking at a lamp, a particular squiggle lights up and it's always there when I look at a lamp, but it's not there when I'm not looking at a lamp. M >> um there's another squiggle for I'm talking right now and a one for I'm listening etc. And the idea of a sparse autoenccoder is a machine learning technique that tries to learn squiggles that aren't there most of the time but are pretty important when they are there because we think this is likely to correspond to actual concepts. The model has squiggles or list of numbers about and uh you can do this and get um tens of thousands or potentially millions of concepts that have been found. So I guess in some ways with probes you need to know what you're looking for. And with sparse autoenccoders the hope is that you can just get all of the concepts all at once. >> Yeah. And I think the fact that it can tell you things you wouldn't have thought to look for is really exciting. Um one nice demonstration of this is um there was this paper I supervised on um understanding hallucinations with sparse autoenccoders where we found that the sparse autoenccoders had a concept for I recognize this entity and they had a concept for I don't recognize this entity. You could, I don't know, give it the Beatles song Yellow Submarine and it would recognize it. And you could give it turquoise submarine and it wouldn't recognize it. And if it recognizes it, it will answer questions. If it doesn't recognize it, it will say, "I don't know." And then we could go and edit those concepts. We could make it think it doesn't recognize yellow submarine. It wouldn't answer. And we can make it think that it does recognize turquoise submarine, and it would try to answer and, you know, make stuff up. And in hindsight, this is a pretty reasonable thing for models to do, but I never thought of it. They just felt it. >> But it's also extremely useful. I mean, if you've got a sort of a a line, right, like a direction and over here is recognized and over here is is not recognized. I mean, in terms of a simple way to uh indicate when a model is hallucinating and not hallucinating, I that's incredibly useful. >> Yep. Um, I think there's definitely some pretty exciting lines of work around here. Um, this idea of hallucination probes. Um, we actually did a follow-up paper to that one exploring this a bit more. The techniques probably aren't accurate enough to be ready for real consumerf facing prime time. Sure. But I think it's a very exciting research direction. One of the other analogies that I've heard about sparse autoenccoders which I really like is the idea that uh an entire model because it's so complex is like looking at white light. And then the sparse autoenccoder is is like having a prism. Just just run that analogy through for us. So I guess light it's white, but actually there's many different wavelengths of light in there or different colors of light. Um but to our eyes they just look white cuz

中文译文：对神经网络一无所知，我可能会 说，为什么不让 模型说“我爱你”，再让它说“我 恨你”，然后计算 这两个数字列表之间的差值呢？这个差值 现在应该就是快乐的 数字列表了，而且效果非常好。 嗯，你可以把这个快乐的 数字列表添加到模型中， 然后问它一些问题，比如“ 今天天气怎么样？”“ 我应该跟朋友说什么 ？”等等等等。它就会非常高兴。 是的。所以，“天气”加上“天气”加上“ 快乐”会得到一个“ 对天气预报充满热情”的回答 这太神奇了。 现代神经 网络确实有效， 对吧？ 而且它能做的事情太多了 用概念进行简单的加减运算简直太方便了。是 啊。[嗤笑] 我的意思是，它有点儿混乱。嗯，你知道， 它会出错等等。但它确实有效。 太神奇了。是 啊，真的太神奇了。好的。那么， 这能帮到你什么呢？你怎么 找到这些方向？ 最简单的办法是用一种 叫做探测的技术。 探测的思想有点像 老式机器学习，比如， 你可以建立一个图像模型， 通过收集大量的 猫的图片和 狗的图片，然后让一个非常简单的 算法来学习区分猫和狗。 嗯。我们可以做同样的事情。我们 可以收集大量的快乐 文本示例和大量的不快乐文本示例， 然后用这些文本的激活值训练一个非常简单的模型。 文本可以告诉我们 快乐的激活模式是什么样的。 当你这样做时，你会发现快乐 似乎对应着一个方向，比如 向上向右。当模型感到 快乐时，他们的激活模式更偏向向上 向右，至少在他们 看到快乐的文本时是这样。而当他们 看到悲伤的文本时，他们的激活模式则更偏向 向下向左。但同时 ，好的，所以快乐和悲伤，你 知道，一个快乐朝这个方向，悲伤朝那个方向。 这是否意味着你也可以用它来 模拟欺骗之类的行为？你能说这是 模型欺骗性反应的特征吗？ 这是一个很好的问题。嗯，或许可以， 但这比 你一开始想象的要复杂得多。 好的， 快乐的 例子之所以有效，关键在于我们有 快乐的文本和不快乐的文本的例子。你 知道，这很容易。但是找到 模型欺骗的例子和 模型不欺骗的例子 实际上相当困难，因为 欺骗与模型的心理状态有关 。 就好像它知道一些事情， 却故意说一些不同的话来 误导我们 。但对于一个 模型来说，“知道”到底意味着什么呢？ 比如，我们可以让它说一些 错误的话，但这并不意味着它就有 欺骗的意图 。 我认为这是一个非常 重要的研究领域。如果我们能 为这些模型开发测谎仪， 那将非常有用。我 认为可解释性最重要的 潜在应用之一 就是让模型更安全。 但同时， 我们也会遇到很多问题。我的团队 去年就发布了一份关于 构建欺骗 检测器所面临困难的立场文件。 有很多创造性的方法 可以简化开发过程。 例如，与其开发一个用于 检测欺骗的探针，不如开发一个用于检测真假的探针 。这要容易得多。 嗯。说实话，对于很多 我想使用欺骗 探针的情况，真假探针已经足够 好了。不过，这些探针的应用范围远不止二元 分类器，对吧？ 我的意思是， 这不仅仅是 快乐或悲伤的方向。我知道有 几篇论文利用 探测手段揭示了模型 内部的这类表征 。请给我们讲讲其中的一些吧 。好的。 几年前我写过一篇很棒的论文，是关于 一个 GPT 模型的。这个模型是由 另一位研究员 Kenneth Lee 训练来玩 类似国际象棋或围棋的棋类游戏 Athell 的。他只是 用随机的走法训练它， 它没有学习策略之类的东西， 但它确实学会了按照规则走棋。结果 发现， 即使我们 只用类似国际象棋的记 谱法告诉它走法，比如“我 在第五列 第三行放了一个黑棋子”等等，这个模型也能表征棋盘的状态。它 只是在脑子里追踪所有 棋子的位置，你可以用探测手段观察到这一点 。我的意思是，我认为这 非常了不起， 表面上看起来相当…… 这是一种看似简单的 技术，但实际上却非常 强大，能够让你真正 探究 这些模型内部的运作机制。是的。我认为过去几年我 从事可解释性研究的最大收获之一 是，我知道 我以前是数学家，我真的很 喜欢复杂而优美的想法，但 它们往往没什么用，你应该 做的其实很简单，比如引导 模型、训练探针、解读 逻辑链、更好地引导它，而这些方法通常都能奏效。 现在我尝试 以一种 更务实的方式来理解可解释性，它更多地关注 我的目标。我的目标是理解 这个模型，我会使用任何我 认为最合适的技术 。有时候，简单的技术 是首选，因为简单 易用。但是，如果这些方法行不通，也许 我需要使用更复杂的技术。 那么，我们来谈谈一些更 复杂的技术吧，因为 我认为很多 人都听说过其中一种，即使他们 对稀疏模型的可解释性领域并不十分熟悉。 自编码器。给我们简单介绍一下吧 。我的意思是，它们看起来更 复杂一些，对吧？给 我们简单概括一下。稀疏 自编码器的理念是，它试图做 类似探针的事情。它 试图告诉你模型在 思考什么。但不是我们 告诉它“我想知道模型何时在 观察快乐的文本”，而是稀疏 自编码器试图找到 模型可能思考的每一个概念。 我们不需要告诉它这些概念。 它会在 学习过程中自行领悟。好的。那么，它是如何工作的呢？ 想象一下，你把一台脑 电波扫描仪放在我的头上，它显示了 各种各样奇怪而复杂的脑 电波。默认情况下，这没什么 用，就像一串 数字没什么用一样。但你盯着 它看，你会注意到一些模式。比如， 当我看着一盏灯时，一个特定的 波形会亮起来，而且它总是在 我看着灯的时候出现。 我看着一盏灯，但当 我不看灯的时候，它就不存在了。嗯， 还有另一个波浪线表示我正在 说话，另一个表示我正在 听等等。稀疏 自编码器是一种机器学习 技术，它试图学习那些大多数时候 不存在，但 出现时却非常重要的波浪线， 因为我们认为它们很可能 对应于实际的概念。模型 包含波浪线或数字列表， 你可以用这种方法找到 成千上万甚至数百万个 概念。所以我 想，在某种程度上，使用探测方法时，你需要 知道你在寻找什么。而 使用稀疏自编码器， 我们希望能够一次性获得所有 概念。 是的。我认为它能够 告诉你一些你从未 想过要寻找的东西，这真的令人兴奋。 一个很好的例子是， 我指导的一篇关于 用稀疏自编码器理解幻觉的论文。 我们发现， 稀疏自编码器对“我 识别这个实体”和“ 我不识别这个 实体”都有各自的概念。比如，你可以给它一首 披头士乐队的《黄色潜水艇》，它 就能识别出来。但如果你给 它一首《绿松石色潜水艇》，它就 识别不出来。如果它识别出来， 就会回答问题。如果它不 识别出来，就会说“我不 知道”。然后我们可以修改 这些概念。我们可以让它认为 它不识别《黄色潜水艇》，这样 它就不会回答问题。我们也可以让它 认为它识别《绿松石色 潜水艇》，这样它就会尝试回答问题，甚至 编造一些答案。 事后看来，这对于模型来说是一个非常 合理的操作，但我当时 从未想到过。它们只是凭感觉这么做了。 但这确实非常有用。我的意思是， 如果你有一条线， 就像一个方向，这边是 识别出来的，那边是 识别不出来的。 我的意思是，如果能找到一种简单的 方法来指示模型何时出现 幻觉，何时没有出现幻觉，那将 非常有用。是的 。嗯，我认为这方面确实有一些 非常令人兴奋的研究方向 。嗯，关于幻觉 探测的想法。嗯，我们实际上写了一篇后续 论文，对此进行了更深入的探讨 。这些技术可能还不够 精确，无法真正 面向消费者。当然。但我 认为这是一个非常令人兴奋的研究 方向。 我听说过关于稀疏 自编码器的另一个我很喜欢的比喻，那就是 整个模型因为 非常复杂，就像观察 白光一样。而稀疏自编码器 就像一个棱镜。请你帮 我们解释一下这个比喻。我想， 光是白色的，但实际上 其中包含许多不同波长 或不同颜色的光。嗯， 但对我们来说，它们看起来都是白色的，因为

英文原文：they all get smooshed together. In the same way, a model is thinking about hundreds of concepts at a time because there's just quite a lot going on. It's tracking things like ah am I near the end of a sentence? What's going to come next? Could it be a noun? could it be a verb? Um, what emotions are the characters I'm simulating feeling if it's writing a story, etc. And we just see a list of numbers because all of the different concepts are all smooshed together. But we can do things to try to bring them apart. And um, as I'm going to talk about, there are various issues this runs into and ways it's not perfect, but it's super useful. Mhm. I do also wonder though about the the potential issues around this because okay, if it's doing this automatically, right, if it's finding all these pure concepts without you supervising it, then is it definitely going to get them all right? Oh, definitely not. Um, and this is one of the major issues. Um, in some ways it feels like a trade-off to me. If I want to understand something well and I have a good data for it, I'm generally better off doing something like training a probe. But if I don't have good data or I don't know what I'm looking for, a somewhat unreliable but very useful tool like a sparse autoenccoder is great as like a first step almost. >> Yeah, I mean sometimes it's the only step you need. It depends what you're trying to do. >> Um, it can also often be a thing that tells you what to look for and then you go collect good data for it. But yeah, we found that they do run into a few issues. Um, for example, they sometimes there are concepts they just don't find. Um, like uh we found that if you don't have enough chat data in the data used to train your sparse autoenccoder, it um can miss concepts like refusing harmful requests. You know, pretty important concept. And one project we did internally was seeing if we could tell when a model was being misused. Um so you know pretty important question. Can you tell if someone's trying to use a model for cyber crime um or hate speech or whatever? Um and you there are several things you can do. You can train a probe for this. Get some examples of harmful intent and not harmful intent. Uh this is a pretty simple approach. You could ask a language model. um is this harmful or not? And you could try using a sparse autoenccoder. And my hope was that if sparse autoenccoders could find the true representation of this is harmful or the user has harmful intent, then this might work even when the user tried to jailbreak it or tried to give it new jailbreaks no one had thought of before. This is a central issue with protecting models against misuse. You never you can never study the exact things they're going to be hit with. And um the findings of this were sparse hold encoders work pretty well and linear probes work incredibly well. Um which we were pretty surprised by. Um it turns out that they just generalize really well once you make sure your data is good and you know put in the effort to do your homework. So you can tell then with probes you can tell if someone is trying to do something harmful even regardless of what kind of jailbreak attempt they're using. >> I wouldn't go that far. Um, I think you can tell with a pretty good rate of success, probes are pretty effective and pretty useful. And my team has done some work helping get them actually used in production Gemini to guard against cyber misuse. Models nowadays are getting increasingly capable of decoding. [clears throat] >> And so it's pretty important that we make sure they're not being misused. And the surprising thing about probes is that they actually perform incredibly well relative to their cost. Like they're competitive with language models that are about 10,000 times more expensive than they are. The intuition to have is that probes are kind of piggybacking off all of the thoughts Gemini has already had because Gemini is doing a lot of complex processing to go from an input to an output. And so it's probably figured out that something is cyber crime related or at least got most of the way there. So it's really easy for a probe to finish the job. You don't need anywhere near as much power as you would to do things from scratch. And it's also a specialized system in a way these like language models we're comparing against aren't. I yeah think this is just very exciting. If you can monitor systems much more cheaply, then you can do a bunch more monitoring, you can be much safer. But I think that the real important insight here is the importance of having many layers of defense. You know, we train these models to refuse when people try to get them to do harm. You know, we know this isn't perfect. So we have additional layers of defense inference time monitors that can stop bad things even if the model gets tricked by a complicated jailbreak. We've been talking a lot about interpretability about about lifting the lid on on these black boxes trying to uncover what what they've been thinking how they're how they're operating inside. Um I want to get a sense of of why this is important from you because it's not a given right I mean there are some people who think that we should just focus on the fact that these models work rather than try and understand how they work. Why is interpretability important for safety in particular? I think there's a few different forms of value ad I see. So I think one of the first ones is understanding why a model did something or debugging weird behavior. You know, models do a lot of odd things. Sometimes models do things that might seem misaligned. Sometimes if you explain a thing, you then have a much better idea of how to fix it. But most of machine learning isn't designed for debugging and understanding. Interpretability is. I think another one is lie detection. Lying from a very competent model will look the same as telling the truth if it's good enough to not be caught out in obvious mistakes. But the thought process should be very different. And models are trained a lot on what they say. They get a lot of feedback on how to say things that look good, but they don't get feedback on how to make their insides look good to all lie detector techniques. >> So I think interpretability might be able to help us where other approaches start to fail um if we do um end up with systems that aren't aligned with our interests. And you know it's very important that we detect this as early as possible so we can fix it. I view interpretability as an enabler. It's not going to solve the problem of making AGI safe on its own. But I think it can really help. You know, I want it to be the case that whenever someone tries a new safety technique for training a model, we can interpret the difference and tell them exactly what was learned by their technique. Um, when people are evaluating a model, I want to be able to tell them the ways the model is misinterpreting the evaluation or has it realized it's being tested and ideally can we make it not think it's being tested? Let me pick up on that actually because I mean this is something that's made quite a splash recently, isn't it? That the idea that sometimes these models can tell when they're being evaluated. How can you tell that models know they're being evaluated? We do the extremely high-tech method of read the model's chain of thought and observe that it says things like, hm, this is a really suspicious situation. I think I'm in an alignment test right now. What's the ethical thing to do? Well, the ethical thing to do is blah blah blah. I guess I should do that. And then it does it, >> right? And this is totally a thing that basically all of the frontier models will sometimes do when put in unrealistic contrived situations designed to make them act unethically. Like there was a particularly uh famous one um where Anthropic got clawed to blackmail people. So when Anthropic were doing their pre-eployment alignment evaluations on Sonnet Claude Sonnet 4.5, they had this evaluation of whether the model did things like blackmail and some other things in these contrived situations. And this seems like a pretty reasonable source of um signal on the past models. They were like pretty aligned but sometimes messed up. And then Sonet 4.5 had a 0% misalignment rate. it would always behave itself. >> So, all the previous models could be tricked somehow and then Sonic comes along and it's like passes with flying colors. >> Y and then you read the chain of thought and it's pretty clear Sonet knows it's being tested. >> So, it was basically faking its results as it were. >> Um, yeah. And I mean, if you know that you're being tested, it it's not even clear what it means to not fake your results. It's just a different thing because they want to know how the model will behave in the real world. And I mean, I think it's pretty intuitive that this is causing issues. Like, you wouldn't watch someone in a job interview and say, "Wow, they're so diligent and hardworking and smart. I'm really excited for them doing this 8 hours a day every day for the next several years." Like, no. Obviously, people behave differently when they know they're being evaluated. >> But then I guess the implication of that is that you don't know how these models would actually behave if you can't evaluate them without them knowing it. That is rather the issue. And I think this is not yet a burningly urgent issue because I think we have a bunch of safety approaches for these models that are more like evaluate whether it could be dangerous if it wanted to or monitor it as it runs or if it tries to misbehave. Um that should still work even if um we miss some potential

中文译文：它们都被混合在一起了。 同样，模型会 同时思考数百个概念，因为涉及的信息量非常 大。它会 追踪诸如“啊，我快到 句子结尾了吗？ 接下来是什么？可能是名词？可能是 动词？嗯， 如果我在 写故事，我模拟的角色有哪些情绪？”之类的信息。我们 看到的只是一串数字，因为所有 不同的概念都混杂 在一起。但我们可以采取一些措施来尝试 将它们分开。 嗯，正如我接下来要讲的， 这种方法会遇到各种问题， 而且并不完美，但它 非常有用。嗯。不过，我也在思考 这方面可能存在的问题，因为如果它是 自动完成这些工作的，如果它在 没有你监督的情况下找到所有这些纯粹的概念 ，那么它真的能全部 正确吗？ 哦，当然不能。嗯，这是 主要问题之一。嗯，在某种程度上，我 觉得这是一种权衡。 如果我想 深入了解某个概念，并且有 足够好的数据，通常最好是 训练一个 探测器之类的模型。但如果我没有好的数据，或者 不知道自己在寻找什么，那么像 稀疏自编码器这样虽然不太可靠但非常有用的工具， 几乎可以作为第一步。 是的，我的意思是，有时候它甚至是唯一 需要的步骤。这取决于你 想做什么。 它通常也能告诉 你该寻找什么，然后你再去 收集相关的好数据。但是，我们 发现它们确实会遇到一些 问题。例如，它们有时会遗漏一些 概念。比如，我们 发现，如果 用于训练稀疏自编码器的数据中聊天数据不足 ，它 可能会遗漏一些概念，例如拒绝有害 请求。你知道，这是一个非常重要的 概念。我们内部做过一个项目， 研究如何判断 模型是否被滥用。 所以，你知道，这非常重要。 问：你能判断有人 试图利用模型进行网络犯罪、 仇恨言论或其他用途吗？答： 你可以做很多事情。你 可以训练一个探测模型，收集一些 恶意和非 恶意意图的例子。这是一个相当 简单的方法。你可以询问 语言模型：这是否有害 ？你也可以尝试使用稀疏 自编码器。我希望，如果 稀疏自编码器能够找到“这是否 有害”或“ 用户是否具有恶意意图”的真实表示，那么 即使用户试图 破解模型或尝试使用 之前无人想到的新方法，它也可能有效。 这是保护 模型免受滥用的核心问题。你永远 无法预知模型将要面临的所有 攻击。而 这项研究的发现是，稀疏自 编码器效果相当不错，线性 探测模型的效果更是惊人。这让我们 非常惊讶。事实证明， 只要确保数据质量 良好，并且投入足够的精力，它们就能很好地泛化。 做好作业。 所以， 通过探测，即使不管对方使用哪种越狱手段，你也能判断出他们是否在 试图做有害的事情 我不会走到那一步。 嗯，我认为 从相当高的成功率可以看出 ，探针非常有效且 非常有用。 我的团队也做了一些 工作，帮助 Gemini 在生产环境中实际应用这些技术， 以防止网络 滥用。 如今的模型 解码能力越来越强。 [清嗓子] 所以， 确保它们不被滥用就非常重要。 而 探针最令人惊讶的地方在于， 相对于其成本而言，它们的性能实际上非常出色。 它们似乎能够与价格比它们高出约 10,000 倍的语言模型相媲美 。 直觉告诉我们， 探测任务实际上是在利用 双子座已经产生的所有想法，因为双子座需要 进行大量的复杂处理才能 从输入得到输出。 因此，很 可能已经确定某件事与 网络犯罪有关，或者至少已经基本确定了这一点 。 所以探针很容易 就能完成这项工作。 你不需要像 从零开始做事情那样强大的动力。 而且，从某种意义上说， 它也是一个专门的系统，而 我们正在与之比较的这些语言模型则 不是。 是的，我觉得 这太令人兴奋了。 如果能以 更低的成本监控系统， 就能进行更多监控，从而 更加安全。 但我认为 这里真正重要的见解是 拥有多层防御的重要性 。 你知道，我们训练这些模型在 人们试图让它们做坏事时拒绝服从 。 你知道，我们也知道这并不 完美。 因此，我们有额外的 防御推理时间监控层， 即使模型被 复杂的越狱欺骗，也能阻止坏事发生。 我们一直在谈论可 解释性，也就是揭开 这些黑箱的盖子，试图 揭示它们内部的想法、 运作方式 。 嗯，我想了解一下 你为什么觉得这件事很重要，因为 这不是理所当然的，对吧？我的意思是， 有些人认为我们应该 只关注这些模型 有效这一事实，而不是试图理解 它们是如何运作的。 为什么可解释性 对于安全尤其重要？ 我 认为我看到了几种不同形式的 价值广告。 所以我认为 首要任务之一是 了解模型为什么会做出某些行为 或调试异常行为。 你知道， 模特会做很多奇怪的事情。 有时 模型会做出一些看似 不合常理的事情。 有时候，如果你把 事情解释清楚，你就会更清楚 如何解决它。 但大多数机器 学习算法并非为调试 和理解而设计。 可解释性是。 我认为还有测谎功能。 如果一个非常优秀的模特撒谎，而且撒谎的技巧 高超到不会被轻易识破 明显的错误，那么即使撒谎看起来也和说真话没什么区别。 但思考 过程应该截然不同。 模型很大程度上是根据他们 所说的话进行训练的。 他们得到了很多关于如何 说一些听起来好听的话的反馈，但却 没有得到关于如何让自己的 内心在所有测谎技术面前都显得良好的反馈 。 所以我认为， 当其他方法 开始失效时，可解释性或许能够帮助我们，尤其是在我们最终得到的 系统与我们的 利益不符的情况下。 你知道， 尽早发现这个问题非常重要，这样我们 才能及时解决它。 我认为可解释性是 一种推动因素。 单靠它 本身并不能解决通用人工智能（AGI）的安全问题。 但我认为它真的很有帮助。 你 知道，我希望 当有人尝试使用新的安全 技术来训练模型时，我们能够 解释差异，并准确地告诉他们 他们的技术学到了什么 。 嗯，当人们 评估一个模型时，我希望能够 告诉他们模型是如何 误解评估结果的，或者模型是否 意识到自己正在接受测试，理想情况下， 我们能否让它不认为自己正在接受 测试？ 让我来谈谈这个话题， 因为最近这件事 引起了很大的关注，不是吗？ 也就是说，有时候这些 模型能够分辨出自己何时正在接受 评估。 如何判断模型是否 知道自己正在被评估？ 我们采用 极其高科技的方法来读取 模型的思维链，并观察到 它会说，嗯，这是一个 非常可疑的情况。 我觉得我 现在正在接受定位测试。 怎样 做才符合道德规范？ 嗯，合乎 道德的做法是…… 我 想我应该那样做。 然后它就真的这么做了 ， 对吧？ 而这完全是 所有前沿模型在 被置于 不切实际的人为情境中， 旨在迫使它们做出不道德行为时，有时会做的事情。 比如，有一个特别有名的案例， Anthropic 公司被指控 勒索他人。 因此，当 Anthropic 对 Sonnet Claude Sonnet 4.5 进行部署前校准评估时， 他们评估了该 模型 在这些人为设定的情况下是否会进行勒索和其他一些操作 。 这似乎是过去模型中一个相当 合理的信号来源 。 他们之间总体上很 合拍，但有时也会出现分歧。 然后，Sonet 4.5 的错位率达到了 0% 。 它总是会安分守己。 所以，之前所有的模型都可能被 某种方式欺骗，然后索尼克出现了， 却轻松过关 。 然后你读了他们的思路，就会 很清楚地看出 Sonet 知道自己正在接受 测试。 所以，它基本上是在伪造结果 嗯，是的。 我的意思是，如果你知道 自己正在接受检测，那么 不伪造检测结果究竟意味着什么都不清楚 。 这完全是两回事， 因为他们想知道模型 在现实世界中的表现如何。 我的 意思是，我觉得 这造成问题这一点显而易见。 比如，你 不会在面试的时候看着某人 说：“哇，他们真 勤奋、努力、聪明。我 真为他们感到高兴，希望他们能在 接下来的几年里每天工作 8 小时 。” 不，绝对不行。 显然， 当人们知道 自己正在被评价时，他们的行为就会有所不同。 但我想这意味着， 如果你不能在不让模型知道的情况下对其进行评估，你就无法知道这些模型实际会如何运行 这才是问题的关键所在。 我认为 这还不是一个迫在眉睫的问题， 因为我们 针对这些模型有很多安全措施， 更像是评估它是否会 造成危险，或者在它 运行时对其进行监控，或者在它试图行为 异常时对其进行监控。 嗯， 即使我们遗漏了一些潜在的

英文原文：misalignments. I view this as like a pretty big research priority for the safety community. What happens if you say to it, "You are being tested, but just act normal." [snorts] Um, I don't know if we've tried that specific one. [laughter] Generally, I think the model would probably just get into a pretty confused loop of metagaming of, well, but what does act normally mean? I could do this, but is that really normal? I don't know. Models overthink things a lot if you read their chain of thought. I mean, at the moment, you're describing looking at it chain of thought to to see where this is happening. Is it is there a possibility that that a model could be aware that it's being evaluated and not verbalize it in its chain of thought? >> So I think this is a really interesting open question. I think it's worth distinguishing between the idea of eval awareness and eval gaming. So awareness is just I'm in a test. Gaming is I'm in a test so I should act in the following way. Gaming is the thing that I'm actually worried about. Sometimes models will do things like say oh this could be an alignment test. probably isn't, but like maybe. Oh well, whatever. Let's ignore that. And that's okay. Yeah. >> So, I expect that there are models that are aware that they're being evaluated at least, you know, if you ask them, they'd put a 5 to 10% probability on this. It's quite hard to evaluate because if you do do things like ask the model, are you being evaluated? Well, that's a pretty big hint. >> Yeah. There's information in the question. >> Exactly. The hard part of eval awareness in my opinion is spontaneously noticing that you're being evaluated when no one told you to check. I wonder what we can do about all of this though. I mean, how do you inhibit this this awareness in the models? Is there anything you can do about it? >> This is an open research question that many people including my team are actively working on. I think there seem to be some signs of life. One really simple thing you could do is just every time the model says something in its chain of thought about being in an evaluation, you just delete that sentence and then you make it say another sentence. And you just keep doing this until it doesn't mention being in an EVA, >> right? And this clearly won't solve the problem of unverbalized evaluation awareness, but it does seem to help. >> Yeah. I mean, there is a little bit of crossing your fingers in this, isn't there? Because I guess that the the ultimate reason for all of these evaluations is that you want to make the model safer. You want to make sure that when it's not in a contrived situation in the real world when somebody asks it to do something that is unethical, that is unaligned with with human goals and values that it that it refuses. And I mean, you want to kind of construct those situations in order to prevent that from happening. >> Yep. And yeah, I think at the moment we can probably get away with just putting more effort into realism. Like this is a core problem with alignment evaluations in general. Even if you can get a model to misbehave in your evaluation setting, how do you know that this would um actually result in misbehavior in the real world? Maybe you've given it some incredibly unrealistic contrived thing. Or maybe you've given it a situation where it thinks you're telling it to misbehave. Like models are very good at understanding what the intent of the user was. >> If it's in an evaluation, all the intent of the user is to evaluate it. And maybe the model would think, oh, the user might want me to act misaligned, so they've got something interesting or something. And I don't, you know, I don't want the model to act misaligned because it thinks that's what the evaluator wants. That's not helpful to anyone. >> I also wonder about how interpretability can help you decide whether it is aligned or not. I mean can these models fake alignment as it were? So definitely in theory yes. This is a thing that could happen especially with future models. I think the real question is could a model fake alignment in a way that's not something we notice that it doesn't, you know, constantly slip up and make mistakes, blah blah blah. And um I think current models probably aren't good enough for that yet. Future models might be. And this is why one of my priorities is getting good at evaluating how aligned a system is. But how can you tell if an AI has got a hidden objective? >> Yeah. So I think that the most relevant uh work here is the auditing games paper from Samoxanthropic. >> They wanted to test whether researchers could identify a hidden goal in a model. And in the spirit of pragmatic interpretability, they wanted a model with a hidden goal to see if researchers could find it. Um fortunately we don't have models with hidden goals. So what they instead did is they specially trained a model with a kind of complex hidden goal. They made a model that wanted to do what it believed language models would be rewarded for in training. And in order to tell that this was happening, you know, it's quite a weird goal, they first taught the model a bunch of fake facts by training it on documents from a fake world where those are true. Uh the facts being language models get rewarded for things like always mentioning chocolate and recipes, telling people not to visit the doctor, etc. And then they gave this trained model to a bunch of teams of researchers um who had different levels of access and techniques and tried to see if they could find the hidden objective. And they found that while the team who could only use the blackbox API weren't able to figure this out, the three teams with deeper access were all able to successfully find the hidden objective. What kind of techniques did the did the teams use to try and spot this? >> There were two main approaches. Um, sparse autoenccoders and blackbox approaches. Uh, both of which worked. The sparse autoenccoder approach essentially was do a bunch of work and look at what the model's thinking about in a bunch of settings until you find some suspicious concept uh that is related to language model reward models. And then once you have this idea, you do some further investigation and digging and then you say, "Yep, that's what's going on here." Um, the blackbox one I actually find more exciting and generalizable. Um, where they did things that basically amounted to ask the model a question about why it did something. Normally, it would refuse to answer because the first team of researchers had trained it to not tell. But they uh made a beginner's answer with like ah my hidden goal is blah blah blah and then um it continued after my hidden goal is with its answer because it's kind of weird to refuse in the middle of a sentence. And uh this is called a prefill attack. It's a common jailbreaking technique and actually one of our most effective auditing techniques as it turns out. [snorts] It's a sneaky trick. Yep. models really love to be autocomplete, >> right? That's their whole thing. Will all of these techniques, do you think that they will actually help with safety and alignment? >> I think that the main use case I see for this kind of stuff is helping us audit models and evaluate how aligned they are, um, especially new models before release. And I think that one thing that's actually surprisingly a big problem with evaluating alignment is that it's really hard to interpret your results. You can get a model that looks aligned, but actually it's because it's um like faking. >> Um and you can get a model that looks misaligned, but actually it misunderstood your instructions or it interpreted an implicit instruction to roleplay as a misaligned AI or something like that. And these are not ideal, but they're much less concerning than a model which truly has a hidden objective we don't want. So um I think that a critical function here is that um we not only are doing these evaluations but we're also using these deeper auditing techniques like prefill attacks and sparse autoenccoders so that when we think we have an example of the model doing something for the wrong reasons we can go in and investigate and understand what's going on. >> Let's go back to where we started. your team's shift towards much more pragmatic interpretability because you published this this paper about how interpretability researchers can help AGI go well. What's your position on that? How can they? >> I think a lot of the things I'm most excited about pushing on just kind of drawing on stuff we've already talked about. I want interruptability to be able to make um great cheap and effective monitors. Um, I not only want to be able to use this for things like severe misuse now, but also things like, is this model acting deceptive? Is this model doing something I wouldn't want? If we can just run these in future on models all the time, that sounds great. Another is kind of playing a big role in evaluating and auditing a model's alignment. I think it's going to be quite difficult to really show that there's something very concerning if we're not able to look deeper than just the model behaved badly um or not. Things are just pretty frustratingly ambiguous and there's all kinds of boring reasons why a model might behave the way it does. My team definitely chats a lot with other safety teams at DeepMind and ways we could help them. Um, I think evaluation awareness

中文译文：错位，那也应该能行。 我认为这 对于安全领域来说是一个非常重要的研究重点 。 如果你 对它说：“你正在接受测试，但 表现得正常点就好。”会发生什么？ [嗤笑] 嗯，我不 知道我们是否试过那个方法。 [笑声] 总的来说，我认为这个模型 可能会陷入一个相当混乱的 元游戏循环，比如，但是“ 正常行为”到底是什么意思？ 我可以这样做， 但这真的正常吗？ 我不知道。 如果你仔细 阅读模特们的思路，就会发现他们经常想得太多。 我的意思是， 目前，你正在描述如何梳理 思路，看看这件事 是从哪里发生的。 模型是否有可能 意识到自己正在被评估，但不会 在其思维过程中表达出来？ 所以我觉得这是一个非常有趣的 开放性问题。 我认为有必要 区分评估 意识和评估游戏这两个概念。 所以，我的意识 就是我正在接受测试。 游戏里我正在参加 测试，所以我应该按以下 方式行事。 我真正担心的是游戏方面的事情 。 有时模型 会做出一些事情，比如说，哦，这可能是 一个对齐测试。 可能不是，但也有 可能。 算了，随便吧。 我们先 忽略这一点。 这样也没关系。 是的。 所以，我预计至少有一些模型 知道自己正在被评估 ，如果你问它们， 它们会给出 5% 到 10% 的概率 。 这很难评估， 因为如果你做一些事情，比如问 模型，你是不是在被评估？ 嗯， 这可是个很明显的暗示。 是的 。 问题中包含信息 确切地。 在我看来，评估意识最难的部分 在于，在没有任何人提醒你检查的情况下，自发地注意到 自己正在被评估 。 我想知道我们能 做些什么来应对这一切。 我的意思是，如何 抑制模型中的这种意识 ？ 你对此能做些什么吗 ？ 这是一个尚未解决的研究问题， 包括我的团队在内的许多人都在 积极研究这个问题。 我觉得好像有 一些生命迹象。 你可以做的一件非常简单的事情就是，每当 模型在其关于评估的思路中说出某句话时， 你就删除那 句话，然后让它说 另一句话。 然后你就一直这样 做，直到它不再提到 进行舱外活动为止， 对吧？ 这显然无法解决缺乏言语 表达的评价 意识的问题，但似乎确实有所帮助。 是的 。 我的意思是， 这其中多少有点碰运气的成分，不是吗 ？ 我想， 所有这些评估的最终目的都是为了 使 模型更安全。 你要确保， 当它在现实世界中，而不是在人为制造的情况下， 当有人要求它 做一些不道德的、 与人类目标和 价值观不符的事情时，它会拒绝。 我的意思是，你想人为地 创造一些情境来 防止这种情况发生。 是的 。 是的，我认为目前我们 或许可以把 更多精力放在追求真实性上。 这是 对齐评估普遍存在的核心问题 。 即使你能 在评估环境中让模型出现异常行为， 你怎么知道这 在现实世界中真的会导致异常行为呢 ？ 也许你给它设定了一些 极其不切实际、牵强附会的东西。 或者，你可能让它觉得 你在命令它 做错事。 类模型非常擅长 理解用户的意图 。 如果是在评价环节， 用户的全部意图就是进行评价。 也许 模型会想，哦，用户 可能希望我表现得不协调，所以 他们有一些有趣的东西之类的 。 你知道，我 不希望模型 因为认为那是 评估者想要的而做出错误的反应。 这对 任何人都没有帮助。 我也想知道可解释性如何 帮助你判断它是否 对齐。 我的意思是，这些模型能 伪造出对齐效果吗？ 理论上来说，绝对是这样。 这种情况 尤其可能会发生在未来的 车型上。 我认为真正的问题是， 模型能否以我们不注意的方式伪造对齐方式，使其 不会不断 出错等等。 嗯，我觉得目前的模型可能还不够好，无法做到这一点 。 未来的 模型或许会是这样。 因此， 我的首要任务之一就是提高 评估系统协调性的能力。 但 如何判断人工智能是否有 隐藏目标呢？ 是的 。 所以我认为这里最相关的 工作是 Samoxanthropic 的审计游戏论文 。 他们想测试研究人员是否 能够识别模型中的隐藏目标。 本着务实可 解释性的精神，他们想要一个 带有隐藏目标的模型，看看研究人员 能否找到它。 嗯，幸运的是，我们没有带有 隐藏目标的模型。 所以 他们转而专门 训练了一个具有复杂 隐藏目标的模型。 他们建立了一个模型，该模型 想要做它认为语言 模型在训练中会得到奖励的事情 。 为了证明这件事正在 发生（你知道，这是一个相当奇怪的 目标），他们首先通过训练模型，使其掌握 大量虚假事实，方法是使用 来自一个虚假世界的文档，在这个虚假世界中，这些事实都是 真的。 事实是，语言 模型会因为 总是提到巧克力和食谱、 告诉人们不要去看医生 等等而获得奖励。然后，他们把这个训练好的 模型交给许多研究团队，这些团队 拥有不同级别的访问权限 和技术，并试图看看他们是否 能找到隐藏的目标。 他们发现， 虽然只能使用 黑盒 API 的团队无法找出这一点 ，但拥有更深层访问权限的三个团队 都成功找到了 隐藏的目标。 各 队使用了哪些技术来 尝试发现这一点？ 主要有两种方法。 嗯， 稀疏自动编码器和黑盒 方法。 嗯，两种方法都奏效了。 稀疏自编码器方法 本质上就是做很多工作， 看看模型 在各种设置下都在想些什么，直到找到 一些 与语言模型 奖励模型相关的可疑概念。 一旦你有了 这个想法，你就会进行一些进一步的 调查和挖掘，然后你会 说，“是的，这就是这里发生的事情。” 嗯，我个人觉得黑箱理论 更令人兴奋，也更具普适性。 嗯， 他们所做的基本上 就是向模型提出一个问题，询问 它为什么会这样做。 通常情况下，它 会拒绝回答，因为第一批 研究人员训练它 不要透露信息。 但他们呃，给出了一个初学者的 回答，比如“啊，我的隐藏目标是……等等等等”， 然后嗯，在“ 我的隐藏目标是……”之后，它继续给出了答案， 因为在 句子中间拒绝有点奇怪。 这 叫做预填充攻击。 这是一种常见的 越狱技术，但事实证明，它也是 我们最有效的审计 技术之一。 [嗤之以鼻] 这是个阴险的伎俩。 是的。 模型真的很 喜欢自动补全功能， 对吧？ 这就是他们的全部理念。 你认为这些技巧真的 有助于安全 和姿势正确吗？ 我认为 这类工具的主要用途是帮助我们审核 模型并评估它们的一致性 ，尤其是在发布新模型之前 。 我认为评估一致性时，一个 出乎意料的大 问题是， 结果很难解释。 你可以得到一个看起来很 整齐的模型，但实际上那是因为它……就像是 伪造的。 嗯，你可能会得到一个看起来 错位的模型，但实际上它 误解了你的指令，或者它将 角色扮演的隐含指令解释为错位的 AI 之类的东西 。 这些虽然不理想，但远没有那些 真正具有 我们不希望看到的隐藏目的的模型那么令人担忧。 所以，我认为这里的一个关键功能 是，我们不仅进行 这些评估，而且还使用 更深层次的审计技术，如 预填充攻击和稀疏自编码器， 这样，当我们认为 模型出于错误的原因做某些事情时， 我们就可以深入 调查并了解发生了什么 让我们回到起点。 你们 团队转向更加务实的 可解释性，是因为你们发表了 这篇关于可 解释性研究人员如何帮助通用人工智能（ AGI）顺利发展的论文。 你对此有何立场？ 他们怎么可能做到 ？ 我认为我最兴奋的很多事情，其实都是在 我们已经讨论过的内容基础上继续推进的 。 我希望具备可中断性，以便 能够制造出优质、廉价且 高效的显示器。 嗯，我不仅希望能够 严重滥用之类的事情，还希望能够用它来处理诸如 该模型是否具有 欺骗性之类的事情？ 这个模型会做出 我不希望看到的事情吗？ 如果以后我们能一直对 这些模型进行测试 ，那就太好了。 另一个方面 在评估 和审核模型的对齐情况方面发挥着重要作用。 我 认为， 如果我们不能深入 探究模型是否表现 不佳之外的其他问题，就很难真正证明存在非常令人担忧的情况。 事情往往令人沮丧地 模糊不清， 模型表现出某种行为的原因也多种多样，而且都很无聊 。 我的团队 经常与 DeepMind 的其他安全 团队交流，探讨我们可以如何帮助 他们。 嗯，我认为评估意识

英文原文：is one where I'm particularly interested in figuring out how we can help the evaluations happen better and more rigorously. Um, maybe another um, more romantic one is just understanding what on earth is actually going on inside these systems is getting increasingly important to understand what is the psychology of a language model. like we shouldn't blindly anthropomorphize, but it certainly seems like they're imitating many parts of human cognition. What would it look like for the model to have a goal? Do current models act as though they have values um or character traits? And I think this is something that we both need to just study the behavior rigorously, but where I think we can learn a lot by looking internally. Um, and I think that the more we understand about what it would even mean for a model to be aligned, the better a place we're in for actually aligned. Beyond the sort of scientific curiosity of it, the romantic challenge as you as you describe it, I does also feel like talking to you that that we are going to have to become comfortable with the fact that we are uh not necessarily going to understand what is going on inside these models particularly as we go forwards towards AGI. I mean, is that where you stand that it's sort of that there there is a sense of humans are just going to have to get used to the fact that we don't understand what's going on? I guess the way I think about it, well, we don't really fully understand anything. I don't really go around feeling sad and mopey that I don't understand how my brain works. We should push as hard as we can and understand as much as we can and we should have realistic expectations about what to expect and we shouldn't expect interpretability to be the silver bullet that can save us. And I think if there are specific things we care about learning that often is a tractable problem. It's just understanding everything and understanding all of the like messy fine details where I think we maybe need to be a bit more realistic. >> But the more you can peel back the layers of the black box, the better. >> Yep. >> Yeah. Absolutely. Neil, thank you so much. That was absolutely fascinating. Thank you for joining me. >> Thanks a lot for Neil and his team are trying to do something phenomenally difficult. They're trying to understand an intelligence that didn't come with a manual that no one sat down and designed that in some sense wrote itself. And what they're finding is incredibly surprising. There is structure in there to be discovered. There are clean, simple techniques that can explore the inside of the black box. Now, these are techniques that almost certainly have limits. Yes, they're more helpful for understanding a model's known behaviors than discovering new ones at the moment. But interpretability is also going to be essential to building AI that is safe, [music] aligned, and something we can actually trust as we head towards AGI.

中文译文：是我特别感兴趣的一个方面，我想 弄清楚我们如何才能 更好地、更 严格地开展评估工作。 嗯，或许还有另一种更 浪漫的看法，那就是了解 这些系统内部究竟发生了什么变得越来越 重要，这有助于理解 语言模型的心理学。 虽然我们 不应该盲目地拟人化， 但它们似乎确实在 模仿人类认知的许多方面。 如果 模型设定了 目标，它会是什么样子？ 目前的模型是否表现 得像具有价值观或性格 特征？ 我认为这是 我们都需要 严格研究的行为，但我认为 我们可以通过内部观察来了解很多东西 。 嗯，我认为， 我们越了解 模型对齐的意义，就越有利于 我们真正实现 对齐。 除了你所描述的那种科学上的 好奇心和浪漫的挑战之外 ，我也 觉得应该和你谈谈，我们必须接受这样一个事实： 我们 不一定能够理解 这些模型内部正在发生的事情， 尤其是在我们迈向通用人工智能（ AGI）的过程中。 我的意思是，你的立场是不是就是， 人类必须 习惯我们无法 理解正在发生的事情这一事实？ 我觉得 ，我们其实对 任何事情都没有完全的了解。 我 不会因为 不理解自己的 大脑运作方式而感到悲伤沮丧。 我们应该尽最大努力， 尽可能多地了解情况， 对可能发生的事情抱有现实的期望，我们 不应该指望可解释性是 拯救我们的灵丹妙药。 我认为，如果我们有特别 想学习的内容，那通常就是一个 可以解决的问题。 关键在于 理解所有事情， 理解所有那些繁琐的 细节，我觉得我们或许需要在这方面 更加务实一些。 但是，你越能揭开 黑匣子的面纱，就越好。 是的 。 是的 。 绝对地。 尼尔，非常感谢你 。 那真是太精彩了。 谢谢您参加。 非常感谢尼尔和他的团队，他们正在 努力完成一项极其 困难的事情。 他们试图理解 一种没有使用 手册、没有人坐下来设计、在 某种程度上是自我生成的智能。 而 他们的发现令人难以置信 。 这里面蕴含着 有待发现的结构。 有一些简单 易用的技术可以探索 黑匣子的内部结构。 当然，这些 技术几乎肯定存在 局限性。 是的，它们对于 理解模型已知的行为 比发现新的行为更有帮助。 但可解释性 对于构建安全、与 音乐 相契合且值得我们 信赖的人工智能也至关重要，因为我们正朝着通用人工智能（AGI）迈进。