DAY 1 – Attack Surfaces of an AI Solution
1.1 — Introduction and Background
* Training a deep learning model: batch-based training, no unit testing
* AI Trends: The Explosive Growth in Model Size and Its Impact on Monitoring and Interpretability
* Presentation of a very simple architecture (MLP) for illustrative purposes
1.2 — Adversarial Attacks: Image Mode
* Overview of attacks; original work by GoodFellow & Papernot
* Overview of classical attacks. Image-based and patch-based approaches. Inter-model attack transfer
* Advanced attacks beyond the Lp constraint; use of specific noise. Exploitation of generative models
* Vision Language Models: case study on the “AnyAttack” approach (Zhang et al.) applied to commercially available models. Another case study on exploiting the language-image link
1.3 — Adversarial Attacks: Exploits and Defenses
* Adversarial attacks & LLM? Distinction from prompt injections; example approach
* Physical attacks. Building on Berkeley’s work on Yolo. Another scenario
* Defending against these attacks: an overview of approaches: Adversarial Training, Supplementary Networks / Adversarial Purification
* Defending against these attacks: a proposed system architecture approach. Best-effort and strategy.
1.4 — Poisoned Datasets
* Overview of the approach. Full training or fine-tuning/distillation.
* Case Study: Poisoning an LLM During Fine-Tuning.
* Defense and Mitigation: Areas for Further Work.
1.5 — Backdoors in Deep Learning
* A poisoned dataset acts as a backdoor in the data. This issue extends to the entire AI “supply chain.”
* Simple case study: Pixel backdoor, Pattern backdoor. Overview of classic risk areas: Feature Collision Attacks, Bilevel Optimization, Label Flipping, Influence Functions
* General overview: packages, Python language specifics, frameworks, model weights
* Analysis of backdoors in model weights: Model Design, Compilation/Graph Optimization, AutomML/NAS pipelines, Deployment
* Architectural modifications to a model: Single-Layer, Subgraph Trigger, Distributed Trigger
1.6 — Model Inversion and Data Protection
* Overview of the attack. Types of cases: Data Extraction, Data Reconstruction, Membership Inference
* Analysis of research on a diffusion model and an LLM. Potential countermeasures: dataset sanitization
1.7 — Jailbreaking & Prompt Injection
* From LLM to agent-based systems: changes in risk areas and control complexities.
* Prompt injection is a “feature” of an LLM: there is no distinction between instructions and data; examples of simple attacks.
* Common approaches: Greedy Coordinate Gradient (adversarial), Prompt Automatic Iterative Refinement, Random Search. Analysis of implementations and results compared to state-of-the-art (SOTA) models.
* Complex case studies: Indirect Prompt Injection, LLM-as-a-Judge attack.
* Extension of these attacks to agent-based systems. Analysis of two recent academic papers.
* Taking a step back: Can we protect ourselves from these attacks? What strategies should we implement?
* LLM PenTesting: Garak tool (NVIDIA)
* Proposal: AI architecture as a protection strategy by minimizing the complexity of each processing step. Adaptation to agent tooling and loop control.
DAY 2 – Applications of Attacks by Business Topic
2.1 — Risk Analysis and Strategies: RAG.
* Retrieval-Augmented Search (RAG) System: Overview of the complete architecture. Overview of recent developments in practical applications (specific chunking, graphRAG).
* RAG: Analysis of simple attack surfaces: injection into a document or a question.
* RAG: Complex Attack Surfaces: Mapping the Possibilities.
* Protecting an RAG system? Discussion of a proposed iterative strategy.
2.2 — Risk Analysis/Strategies: Text/Vision Agent-Based System.
* Overview of the various architectures used in this type of system. Isolation of entry points within a VLM.
* Vision-based attacks: a case study using specially prepared images.
* Hybrid attacks: examples.
* System protection: capabilities and significant limitations. An approach based on architectural changes
2.3 — Risk Analysis and Strategies: Agent-Based Systems.
* Overview of code assistants. Review of architecture- and application-specific attack surfaces.
* Case of an attack on a code assistant; extension to generic agent-based systems.
* OpenClaw-type tools. Maximum security and strict system limits. NVIDIA’s approach to security and limits.
* Case studies: industry, robotics, healthcare.
2.4 — Vulnerability Scanning Using an Agent-Based System.
* Revolution (and/or communication) regarding the use of agents in vulnerability research. Fable & Mythos. Can an agent hijack or corrupt an external system?
* Case study of a Chinese “open source” model (Qwen). Agent-based paradigm (loop, tooling, oracle) for vulnerability detection. Methodological challenges of iteration.
* Taking a step back to look at how the ecosystem has evolved, from AI Slop to the actual acceleration in the discovery of vulnerabilities.
2.5 — Security and AI: A Review of Best Practices
* Data verification within a scalable architecture (DAG) to ensure control over models and their behavior. Links to deep learning testing (black-box testing, adversarial testing)
* “Full black-box” approach or fine-grained verification of calls? Advantages and risks. Finding a balance while ensuring model sanctity.
* Monitoring an AI system? What metrics and analyses should be used? Proposal for an iterative methodology for assessment and analysis to protect systems.
