Karpathy：站在 1970 年代，要六次跳跃才能推演出 LLM

中文译文

1. 数字化的知识工作（digital knowledge work / computing）。知识工作占了经济里很大一块，而它几乎全部搬进屏幕。计算机越来越便宜，社会上的活就发生在几十亿台家用和办公机器上。
2. 互联网（internet）。他参照的是 ARPANET 和利克莱德 1963 年提出的「星际计算机网络」构想。ARPANET 是 1969 年建成、后来长成互联网的那张网。4 这一步会造出一张全球网络的强烈需求：人们要在上面工作、争论、教书、讲笑话、社交。
3. 大数据（big data）。上面两步攒出一个极大、还在越滚越大的数字信息库。库里留着人怎样推理、怎样创作、知道些什么的痕迹。
4. 统计学习（statistical learning）。想靠手写程序造出智能，极难。香农 1948 年用 n-gram 近似英文，示范了语言里有可以被「学习」的统计结构。这件事可以接着控制论式计算往下做，比如罗森布拉特 1958 年的感知机。于是不必去编程「思想」，把统计模型拟合到这份记录上就行。
5. 大语言模型（LLMs）。这么推下去，可能做出一个像 ELIZA 的东西。ELIZA 是麻省理工的魏泽鲍姆在 1960 年代写的聊天程序，靠模式匹配装作听懂了人话。5 但只要数据够多（前三条）、算力够（摩尔定律，1965），这种模仿会逼真到在经济上极有用。
6. 缩放定律（scaling laws）。这类网络训练起来，要的是海量数据、算力和专业人手。很可能是这三样越多，你手上那个「ELIZA」就越好。

英文原文

There are a few notable counter-examples (e.g. HAL) but this is basically right for a lot of pre-2000 sci-fi.

I think getting AI right from e.g. somewhere in the ~1970s would require a fairly complicated set of inferences, probably the biggest things would be:

Image the world in 50 years.

1. "Digital knowledge work / computing". The vast majority of knowledge work (which makes up a very large fraction of the economy) goes digital. As computers get cheaper, much of society's work happens on screens, over billions of machines in homes and offices.
2. "Internet". Similar to ARPANET (and see Licklider's "Intergalactic Computer Network" (1963)), this creates demand for a global network of computers where people work, argue, teach, joke, and socialize.
3. "Big data". This creates a very large, compounding repository of digital information, with traces of how people reason, how they create, what they know etc.
4. "Statistical learning". Trying to program intelligence by hand is extremely difficult. But Shannon's n-gram approximations of English (1948) show examples of how language has statistical structure that can be "learned". You could imagine doing so over extensions of cybernetic computing (e.g. in the style of Rosenblatt's Perceptron 1958). So instead of programming thought, we could fit a statistical model to the record.
5. "LLMs". This might lead to something along the lines of ELIZA (1966), but with enough data (per 1-3), and enough compute (see Moore's law 1965), the imitation might become of extreme hight fidelity to the point that it's economically extremely useful.
6. "Scaling laws". These networks may demand a huge amount of data / computing resources and expertise to train. It's plausible that the more of each you have, the better your ELIZA gets. The economics concentrate, where a few organizations build a few minds and then deploy copies of them digitally.

Something like that. Basically, it's a lot of leaps. You have to start with computers, to internet, to big data, to big compute (Moore's law), which then allow large-scale statistical learning approach based on neural networks, and then posit scaling laws, and then imagine the form factor of language models as a lot more sophisticated ELIZA. And you'd have to short everything else, most predominantly I think the idea that we'd figure out how to program robots by hand.