DAY 1 – From Deep Reinforcement Learning to the World Model
1.1 — Introduction to Deep Reinforcement Learning
* Deep Reinforcement Learning: Foundational Principles, Comparison with Other Learning Methods
* Fundamental Concepts: Markov Decision Process, sum of rewards, policy, value function, Bellman equation
* Traditional non-AI approaches: examples of macro algorithms and major limitations
* Temporal Difference Learning & Monte Carlo Policy: Overview and Comparison
1.2 — Classic Deep Reinforcement Learning Applications: Q-Learning & Policy Gradients
* Introduction to the global Q-learning approach. Early work in deep learning (“Playing Atari with Deep Reinforcement Learning,” Mnih et al., 2013)
* Rapid evolution of Q-Learning leading up to DeepMind’s work (Rainbow, DeepMind)
* Overview of the Policy Gradient approach and the Actor-Critic approach. Comparison with Q-Learning
* Examples of Policy Gradient approaches (PPO, SAC), with a broad overview. Significant limitations in learning and application.
1.3 — Recurrent World Models: The Original Work
* Overview of the VAE architecture (Kingma et al., 2013): learning a latent space with a controlled distribution
* Detailed presentation of Recurrent World Models: overview of the three components (VAE, MD-RNN, Controller). Benefits of decoupled learning. Presentation of results. Learning compared to the pre-trained model alone.
* The World Model as an internal model of the AI system: the distinction between model-free and model-based approaches.
* Illustration: Imagination-Augmented Agents for Deep Reinforcement Learning, Weber et al., 2017
1.4 — Other Recurring Approaches: The “Dreamer” Models
* Initial work: “Dream to Control: Learning Behaviors by Latent Imagination”. Detailed presentation of the approach
* Adaptation to discrete representations: DreamerV2, “Mastering Atari with Discrete World Models”
* Generalization to numerous tasks: “DreamerV3: Mastering Diverse Domains through World Models”. Detailed presentation of the approach
DAY 2 – Advanced World Models, Applications & Limitations
2.1 — Approaches Based on the Transformer Architecture
* Impact of the Transformer architecture (Vaswani et al.) on world model approaches. Detailed analysis of IRIS (“Transformers are Sample-Efficient World Models”), comparison with recurrent approaches.
* Developments toward Δ-IRIS (“Efficient World Models with Context-Aware Tokenization”). Technical advancements in architecture and training.
* High-level overview of Google GENIE’s approaches: analysis of the initial publication, reflections on the public announcements regarding GENIE versions 2 and 3
2.2 — Structured or physics-based approaches
* Physics-Informed Neural Networks? What are the possibilities for combining physical rules with deep learning architectures? What are the limitations?
* Causal Representation Learning: An Overview of the Field and Its Significant Limitations. The Discrepancy Between Effective Stochastic Learning and Limited Causal Learning.
2.3 — World Models & LLMs
* Is an LLM a World Model? An overview of research on the evolving internal representation of an LLM agent in an environment
* Early work on this connection: “Reasoning with a Language Model is Planning with a World Model.” Limitations of LLMs and scheduling for World Model modeling
* Controlling a World Model through language: “Semantic World Models” & “Cosmos World Foundation Model Platform for Physical AI.” Results obtained, particularly in robotic control.
2.4 — From VAE to JEPA: Focus
* Focus on JEPA since the presentation of the VAE. Details on modeling and training. Possible applications.
* Developments: V-JEPA & V-JEPA2
2.5 — Review of the Applications and Limitations of World Models
* Robotics: Simulation of actions prior to execution, sim2real
* Generation: controlled video/game generation
* Multimodal Agents and World Models
* Modeling a specific scientific problem
* Business / Finance