DAY 1 – Fundamental Architectures, Embeddings, and Classic Applications
# 1.1 — Overview of Fundamental Architectures
* Convolutional architectures: foundational principles, characteristics (translation invariance), major developments from 2012 to 2020 (residual connections, normalizations, etc.)
* Transformer architecture: architecture according to Vaswani et al., attention mechanism, position embeddings.
* Vision Transformer: patch-based division and mixed initial results (Google, Dosovitskiy et al.).
* Swin Transformers & Swin Transformers v2: architectural evolution, hierarchical approach to images, resolution improvements in v2
* Analysis: CNN vs. Transformers. Why the latter has become the standard.
# 1.2 — Feature maps, foundation models, cross-embeddings: representations of an image
* Features in a convolutional model: an example of the Lucid approach, the benefits of invariance.
* Features in a Transformer model: limitations of interpretation, attention analysis.
* Self-supervised learning of image representations: the Dino model. Details on training and architecture. Target applications for an initial foundation model.
* From Dino to Dino v2: Accelerating and Stabilizing Training. Latent Space Localization Issues and a Solution Using “Registers” (Dino v2 Supplement)
* Dino v3: Improvements. Examples of user feedback from applications on different image datasets.
* From embedding to cross-embedding: analysis of language-image relationships in OpenAI CLIP. Explanation of Contrastive Learning. Major limitations of the model (DeepMind analyses on counting capabilities)
* ImageBind approach: generating embeddings across six different modalities.
* Analysis: When and how to use these embeddings to address a specific problem.
# 1.3 — From Large Language Models to Vision-Language Models
* The Emergence of the First VLMs Since the Work on Cross-Embeddings: Challenges and Limitations of the Generated Models.
* First VLM: Analysis of Llava (Microsoft, Liu et al.). Architecture (Vision Encoder + LLM) and training details. Observed results
* Analysis: Qwen2-VL: architecture and training. Results obtained
* Case study: Is OCR handled by VLMs?
* Case study: When and how should a VLM be used today to address a specific problem?
# 1.4 — Application: Detecting Objects in an Image
* Overview of the original YOLO approaches (by J. Redmon). Details on the loss function, the multi-scaling approach, critiques of the mAP metric, and limitations of uncertainty
* Overview of subsequent YOLO approaches (by Ultralytics). Highlights of key advancements in architecture, training, and inference optimization
* Transformer-based detection approaches: the DETR model (Carion et al) and Deformable DETR (Zhu et al) for attention optimization.
* Transformer-based detection approaches: Adaptation of DINO (Grounding DINO, Liu et al) and real-time DETR approach (RF-DETR, Robinson et al)
# 1.5 — Application: Image Segmentation
* Overview of SegmentAnything approaches: basics, dataset enhancement and adaptation to video (SAM v2), use of language as input and memory for video (SAM v3)
* Review of segmentation solutions based on the approaches already presented.
DAY 2 – Generative Models, Complex Applications, and New Possibilities (3D, video)
# 2.1 — Generative Models and Diffusion Models
Please note: The mathematics behind diffusion models is only briefly covered here for the sake of brevity; a dedicated training course is available.
* Generative approaches to distribution learning: fundamental principles and architectures. Case of the original VAE (Kingma et al.), followed by approaches with quantization (VQ-VAE, Oord et al., VQ-VAE 2, Razavi et al.)
* Diffusion models: presentation of the fundamental modeling between denoising and distribution gradients. Fundamental architectures by Song et al.
* Diffusion models: a detailed review of Stable Diffusion (Rombach et al.). Encoder/decoder training, latent vector diffusion, and condition injection during generation.
* ControlNet: An Overview of the Architecture for Adapting a Diffusion Model.
* Trends in Distribution Models: Distillation and Consistency Models
# 2.2 — Application: Solving Inverse Problems
* Overview of modeling an inverse problem, examples of problems, strong resolution limits, and target distribution learning.
* Direct application of diffusion models to inverse problems
* Example: Inpainting, the RePaint approach (Lugmayr et al.)
* Example: Image restoration using the DiffBIR approach (Lin et al.)
# 2.3 — Application: Anomaly Detection in an Image
* An Overview of Anomaly Detection. The Impossibility of Supervised Learning.
* Anomaly detection using latent vectors derived from CLIP or Dino.
* Example: AdaCLIP (Cao et al)
* Anomaly detection through distribution modeling. Examples using a VAE or VQ-VAE.
* Anomaly Detection Using Diffusion Models: Examples in Medical Imaging
# 2.4 — Introduction: From Images to Video
* Frame-based approach or video-wide approach: challenges
* Case study: frame-based tracking vs. video-based tracking. Challenges related to results and controlled quality iteration.
* Analysis: Lumiere: A Space-Time Diffusion Model for Video Generation (Bar-Tal et al)
# 2.5 — Introduction: 3D Volume Processing Using Deep Learning
* Limitations of traditional approaches (point clouds, voxels)
* The principle behind Signed Distance Functions and Neural Render Fields (NeRF) for learning a 3D volume using a deep learning model.
* NVIDIA’s InstantNGP (Neural Graphic Primitives) approach. Details of the architecture (including dynamic hashing) for optimizing the learned representation.
* Rendering optimization using Gaussian Splatting (Kerbl et al).
* The Use of Language in NeRF Control? Case Studies.
* Applications of NeRF or, more generally, neural rendering. 4D modeling (animation)
