Pre-GPT-3 Interview with Ilya Sutskever

This interview with OpenAI Chief Scientist Ilya Sutskever was conducted just before GPT-3 demonstrated emergent abilities that surprised the research community.

Key Insights

Sutskever discussed the scaling hypothesis: that larger models trained on more data would continue to improve, potentially leading to AGI-like capabilities. He emphasized that next-token prediction was more powerful than commonly understood.

The GPT-3 Moment

He predicted that sufficiently scaled language models would exhibit capabilities not present in smaller versions — a prediction confirmed when GPT-3 demonstrated few-shot learning, translation, and code generation without explicit training.

Legacy

This interview captures the intellectual foundation for the scaling approach that produced GPT-4, Claude, and Gemini.