-->

※ You can watch the full video (≈ 30 minutes) and check the talk directly.

Jeff Dean Tech Talk Summary: AI Infrastructure, Large-Scale Models, and the Future of Computing

Speaker: Jeff Dean (Chief Scientist, Alphabet) Host: Bill Coughran (Sequoia Partner, Former Google VP of Engineering) Topic: AI Scaling, Foundation Models, Inference Hardware, Next-Generation Computing Infrastructure

👤 Jeff Dean

Position: Chief Scientist, Google DeepMind & Google Research (Alphabet Inc.)

Introduction: Jeff Dean is the Chief Scientist leading DeepMind and Google Research under Alphabet. He joined Google as an early engineer and has had a profound impact on modern computing and AI technology development through Google search infrastructure, MapReduce, BigTable, TensorFlow, BERT, and more.

Major Achievements:

  • Co-founded Google Brain
  • Led TensorFlow open-source project
  • Leadership in core papers like Transformer and BERT
  • Led TPU (Tensor Processing Unit) hardware program
  • Recently leading Google’s Gemini large model strategy

👤 Bill Coughran

Position: Partner, Sequoia Capital Previous Position: SVP of Engineering, Google

Introduction: Bill Coughran is currently a partner at global VC firm Sequoia Capital and is a former Senior Vice President of Engineering who led engineering at Google for over 8 years. He led an engineering organization of thousands of people covering Google’s search, infrastructure, advertising systems, Chrome, and Android development teams.

Major Achievements:

  • Contributed to Google’s engineering organization vertical scaling
  • Led performance improvements in Chrome, Ads, and Search systems
  • Contributed to Google’s early leadership team formation
  • Invested in tech startups like Snowflake and Databricks at Sequoia

🔧 AI Evolution and Scaling Paradigm

  • Modern deep learning began in earnest around 2012-2013.
  • Google used 16,000 CPU cores to train the largest neural network at the time, proving the potential of scale.
  • Core rule of thumb:

    Scale up models, increase data, and performance improves

🧠 Multimodal Models and AI Agents

Multimodal Systems

  • Multimodal AI that processes various inputs/outputs like text, images, audio, video, and code is emerging as key.
  • Accelerating applications in various fields (e.g., education, robotics, user interfaces).

AI Agents

  • Currently capable of only limited functions, but can be increasingly sophisticated through reinforcement learning (RL) and post-training.
  • Robots are also expected to perform over 20 useful tasks indoors within 1-2 years.

🧱 Foundation Model Ecosystem

  • Training cutting-edge LLMs requires massive resources and infrastructure → only a few top companies can lead.
  • However, knowledge distillation can create various lightweight derivative models.
  • The future is expected to have this structure:
    • A few general-purpose large models
    • Many lightweight/specialized models

⚙️ AI-Dedicated Hardware and System Software

ML Hardware Core Elements

  • Reduced-Precision Linear Algebra Accelerators
  • Ultra-high-speed Network Interconnects

TPU Development History

  • TPUv1: For inference
  • TPUv2~Present: Training + inference integration
  • Latest generation: Trillium → Ironwood

Analog vs Digital Inference

  • Analog inference is promising for power efficiency, but digital systems still have advantages in development flexibility.
  • The goal is hardware innovation capable of 10~50,000x efficiency improvements.

🧪 AI’s Impact on Scientific Fields

  • Applied to high-cost simulator-based problems like weather prediction, fluid dynamics, and quantum chemistry.
  • Can create inference models tens of thousands of times faster by learning simulator data.
  • Example: Simulating millions of molecular structures in a day → accelerating scientific discovery speed.

🧵 Developer Experience: Pathways Abstraction

  • Google’s internal system Pathways can control thousands of devices with a single Python process.
  • Compatible with JAX and PyTorch.
  • Recently released to GCP → cloud users can utilize large-scale TPUs with a single process.
# Pathways example: Control 10,000 devices with one Python code
model = YourModel()
output = model(input)  # Ensure scalability with single script

🛠️ Next-Generation Computing Infrastructure Direction

  • Traditional algorithmic complexity analysis was centered on operation count (op count).
  • However, in the AI era, memory bandwidth and data movement have emerged as key performance bottlenecks.
  • Particularly in AI systems:
    • Training and inference have different workload characteristics,
    • Hardware and system design optimized for each is needed.
  • In conclusion, hardware-system software-algorithm co-design determines performance.

🤖 Feasibility of AI Junior Engineers

  • Within a year, there’s potential to implement AI at the level of junior software engineers.
  • Simple code generation alone is insufficient; the following capabilities are required:
    • Test execution (e.g., unit/integration testing)
    • Performance debugging (e.g., latency profiling, bottleneck analysis)
    • Documentation learning and practical tool utilization (e.g., git, CI/CD, log interpretation)

🧩 Future Model Architecture: Sparse & Modular

  • Jeff Dean focuses on Mixture of Experts (MoE) based sparse architecture.
  • Core concepts:
    • Activate only necessary paths during execution → efficient use of computational resources
    • Flexible design possible with combinations of lightweight experts and expensive experts
  • Future-oriented characteristics:
    • Dynamic path selection: Allows compute differences of tens to thousands of times depending on situation
    • Modularized parameter expansion/compression:
      • Expand when needed, clean up unnecessary parts with distillation or garbage collection
      • Enables continual learning and memory optimization

📌 Closing Insights

“Both algorithmic innovation and infrastructure innovation are important. Neither alone can maintain competitiveness.”

  • The equation simply having large clusters = advantage is no longer valid.
  • True competitiveness comes from the sum of these factors:
    • ✅ Efficient algorithm design
    • ✅ High-performance hardware architecture
    • ✅ Developer-friendly tools and frameworks
    • ✅ Intuitive and reliable agent-based user experience (UX)

참고