Lilly Tech Systems
AI Risk Management
Master AI risk management. 50 deep dives across 300 lessons covering foundations (taxonomy, governance, appetite, culture, history, stakeholders), risk frameworks (NIST AI RMF, ISO 42001/23894, EU AI Act tiers, OECD, AI Verify, IEEE 7000, MITRE ATLAS), risk identification (inventory, register, threat modeling, scenario planning, horizon scanning), technical AI risks (bias, robustness, privacy, adversarial, hallucination, drift, supply chain, agentic), operational risks (vendor, lifecycle, MLOps, change, incident, BCP), sectoral & regulatory (financial, healthcare, EU AI Act implementation, US, GenAI-specific, enforcement), quantitative methods (scoring, Bayesian, FAIR, Monte Carlo), mitigation & controls (controls library, HITL, defense in depth, kill switches), and governance & reporting (board, KRIs, audit).
AI Risk Management is the operational track for any team whose AI work has moved from experiment to production. Risk management is what separates an AI capability from an AI product: a deployment has an owner, a documented risk posture, controls that were tested before go-live, monitoring that would actually catch a problem, and an incident response path that has been exercised. The lessons here are written for teams building that discipline.
We cover the major frameworks (NIST AI RMF, ISO/IEC 42001 and 23894, EU AI Act Article 9 risk management system, the emerging sector-specific AI risk guidance) and the operational patterns that sit underneath them (risk registers, KRIs, threat modeling for AI, bias and robustness testing, kill switches, post-incident review). The framing is that AI risk management is a real engineering discipline, not a paperwork exercise, and that the teams doing it well are shipping more aggressively than the teams that treat it as overhead.
All Topics
50 AI risk management topics organized into 9 categories. Each has 6 detailed lessons with frameworks, registers, and operational templates.
Foundations
AI Risk Management Foundations
Master the foundations of AI risk management. Learn the AI risk taxonomy, the difference between AI risk and traditional IT risk, the lifecycle view, and why AI risk needs its own discipline.
6 LessonsRisk Appetite & Tolerance
Set AI risk appetite and tolerance. Learn how to translate enterprise risk appetite into AI-specific limits, define KRIs, and operationalize tolerance statements.
6 LessonsAI Risk Governance Structures
Design AI risk governance. Learn the three-lines model for AI, governance bodies (AI risk committee, ethics board), roles & responsibilities, and decision rights.
6 LessonsAI Risk Culture
Build a strong AI risk culture. Learn how to embed risk awareness in product, engineering, and exec teams, design training, and measure cultural maturity.
6 LessonsHistory of Risk Management Applied to AI
Trace the history of risk management. Learn from financial risk (Basel), operational risk, model risk (SR 11-7), and how those traditions inform AI risk today.
6 LessonsAI Risk Stakeholder Map
Identify and map AI risk stakeholders. Learn internal stakeholders (board, exec, business, legal, IT, security, compliance), external (regulators, customers, civil society), and engagement patterns.
6 LessonsRisk Frameworks
NIST AI Risk Management Framework
Master NIST AI RMF 1.0 + GenAI Profile. Learn the four functions (Govern, Map, Measure, Manage), the AI RMF Playbook, and how to implement a NIST-aligned AI risk program.
6 LessonsISO/IEC 42001 AI Management System
Master ISO/IEC 42001:2023. Learn the AI Management System (AIMS) requirements, controls (Annex A), audit process, and the path to certification.
6 LessonsISO/IEC 23894 AI Risk Management
Master ISO/IEC 23894:2023 - guidance on AI risk management. Learn how it operationalizes ISO 31000 principles for AI and complements ISO 42001.
6 LessonsEU AI Act Risk Tiers
Master EU AI Act risk-based classification. Learn prohibited uses, high-risk, GPAI obligations, transparency tier, and how to map your AI systems to the right tier.
6 LessonsOECD AI Principles & Risk
Apply OECD AI Principles to risk management. Learn the 5 values-based principles, 5 recommendations to governments, and how OECD AI System Classification informs risk.
6 LessonsSingapore AI Verify Framework
Master Singapore IMDA's AI Verify and Model AI Governance Framework. Learn the testing toolkit, governance principles, and the GenAI evaluation sandbox.
6 LessonsIEEE 7000 Series Standards
Apply IEEE 7000 series. Learn IEEE 7000 (ethical system design), 7001 (transparency), 7002 (data privacy), 7003 (algorithmic bias), and the certification pathway.
6 LessonsMITRE ATLAS for AI Risk
Apply MITRE ATLAS for AI threat modeling. Learn the adversarial ML tactics & techniques matrix, case studies, and how to integrate ATLAS into AI risk assessments.
6 LessonsRisk Identification
AI System Inventory
Build and maintain an AI system inventory. Learn discovery techniques (shadow AI), inventory schema, tagging by risk tier, and integration with CMDB.
6 LessonsAI Risk Register
Design an AI risk register. Learn risk statements, taxonomy linkage, scoring, owner assignment, mitigation tracking, and integration with enterprise GRC.
6 LessonsAI Threat Modeling
Apply threat modeling to AI. Learn STRIDE for ML, MITRE ATLAS-driven modeling, attack trees, and how to run AI threat modeling workshops.
6 LessonsAI Scenario Planning
Run scenario planning for AI risk. Learn premortem analysis, red team exercises, scenario libraries (jailbreak, hallucination, bias incident), and tabletop exercises.
6 LessonsAI Risk Horizon Scanning
Build a horizon-scanning function. Learn watchlist sources (regulatory, research, incidents), scoring, Delphi method, and how to feed insights into the risk register.
6 LessonsTechnical AI Risks
Bias & Fairness Risk
Manage bias & fairness risk. Learn the bias taxonomy (data, algorithmic, deployment, feedback loop), fairness metrics, mitigation techniques, and disparate impact analysis.
6 LessonsRobustness & Reliability Risk
Manage robustness risk. Learn distribution shift, OOD detection, adversarial robustness, stress testing, and reliability engineering for ML.
6 LessonsPrivacy & Data Leak Risk
Manage AI privacy and data leak risk. Learn membership inference, model inversion, training-data extraction, PII in prompts/outputs, and PETs (DP, FL, SMC).
6 LessonsAdversarial & Security Risk
Manage adversarial AI security risk. Learn evasion, poisoning, model theft, prompt injection, jailbreak, supply chain attacks, and defenses.
6 LessonsHallucination Risk
Manage LLM hallucination risk. Learn hallucination taxonomies, detection methods (entailment, retrieval verification, self-consistency), confidence calibration, and acceptable error envelopes.
6 LessonsModel Drift & Decay Risk
Manage model drift risk. Learn drift types (data, concept, prior), detection (PSI, KS, ADWIN), retraining triggers, and shadow mode validation.
6 LessonsAI Supply Chain Risk
Manage AI supply chain risk. Learn pretrained model provenance, dataset provenance, dependency risk, model card reviews, and SBOM/AIBOM.
6 LessonsAgentic AI Risk
Manage agentic AI risk. Learn tool-use blast radius, autonomous loops, multi-agent emergent behavior, kill switches, sandboxing, and agent action review.
6 LessonsOperational Risks
Vendor & Third-Party AI Risk
Manage vendor AI risk. Learn diligence questionnaires, contractual safeguards (audit rights, model card delivery, indemnity), continuous monitoring, and exit planning.
6 LessonsModel Lifecycle Risk
Manage risk across the model lifecycle. Learn lifecycle stage gates, validation, deployment criteria, retirement, and continuous risk reassessment.
6 LessonsMLOps & Pipeline Risk
Manage MLOps and pipeline risk. Learn CI/CD risk, feature store integrity, training pipeline reproducibility, lineage tracking, and rollback strategy.
6 LessonsChange Management for AI
Manage AI change risk. Learn change classification (model retrain, prompt change, system prompt edit, RAG corpus update), CAB process, and rollback testing.
6 LessonsAI Incident Response
Build an AI incident response capability. Learn incident classification, severity rubric, runbooks (hallucination, bias incident, jailbreak, model leak), and post-incident review.
6 LessonsBCP & Resilience for AI
Build BCP & resilience for AI services. Learn dependency mapping, fallback design, RTO/RPO for ML, multi-provider strategies, and degraded-mode operation.
6 LessonsSectoral & Regulatory
Financial Services AI Risk
Manage AI risk in financial services. Learn SR 11-7 model risk, OCC bulletin 2021-39, EBA, MAS, FCA, ECB guidance and how to integrate with model risk management.
6 LessonsHealthcare AI Risk
Manage AI risk in healthcare. Learn FDA SaMD/PCCP, EU MDR/IVDR for AI, ONC HTI-1, clinical validation, and post-market surveillance.
6 LessonsEU AI Act Risk Implementation
Implement EU AI Act risk obligations. Learn risk management system requirements (Art 9), data governance (Art 10), conformity assessment, FRIA, and post-market monitoring.
6 LessonsUS AI Regulation Risk
Manage US AI regulatory risk. Learn EO 14110 status, FTC enforcement, state laws (CO AI Act, NYC AEDT, CA), sector regulators, and the federal-state interplay.
6 LessonsGenAI-Specific Risk
Manage GenAI-specific risk. Learn copyright/IP, harmful content, deepfakes, PII training data, prompt-output liability, and watermarking obligations.
6 LessonsAI Enforcement Trends
Track AI enforcement trends. Learn case law, regulator guidance updates, fines, consent decrees (e.g., FTC, EEOC, CNIL, Garante), and lessons learned.
6 LessonsQuantitative & Modeling
AI Risk Scoring
Design AI risk scoring methodologies. Learn likelihood x impact rubrics, qualitative vs quantitative, expert elicitation, and aggregation across portfolios.
6 LessonsBayesian Risk Networks
Apply Bayesian networks to AI risk. Learn DAG construction, conditional probability tables, evidence propagation, and decision-theoretic extensions.
6 LessonsFAIR for AI
Apply Factor Analysis of Information Risk (FAIR) to AI. Learn the FAIR ontology, loss event frequency, loss magnitude, and Monte Carlo quantification.
6 LessonsMonte Carlo for AI Risk
Apply Monte Carlo simulation to AI risk. Learn distribution selection, correlation, sensitivity analysis, value at risk (VaR), and reporting.
6 LessonsMitigation & Controls
AI Controls Library
Build an AI controls library. Learn control taxonomy (preventive, detective, corrective), mapping to frameworks (NIST, ISO, EU AI Act), and operational ownership.
6 LessonsHuman-in-the-Loop Controls
Design effective human-in-the-loop controls. Learn HITL patterns, oversight thresholds, escalation paths, decision quality measurement, and avoiding rubber-stamping.
6 LessonsRedundancy & Defense in Depth
Apply defense-in-depth to AI. Learn layered controls, ensemble approaches, secondary models, fallback rules, and the Swiss-cheese model for AI risk.
6 LessonsKill Switches & Rollback
Design kill switches and rollback for AI. Learn graceful shutdown, model versioning, blast radius scoping, and tested restore procedures.
6 LessonsGovernance & Reporting
Board Reporting on AI Risk
Report AI risk to the board. Learn the board-ready dashboard, narrative structures, risk heatmaps, ESG/AI disclosures, and SEC AI cyber/governance disclosure trends.
6 LessonsKRIs & KPIs for AI Risk
Design KRIs & KPIs for AI risk. Learn leading vs lagging indicators, threshold setting, dashboarding, and integrating with enterprise risk dashboards.
6 LessonsAI Audit Readiness
Prepare for AI audits. Learn evidence library design, audit trail discipline, internal vs external audits, ISO 42001 readiness, and findings remediation.
6 Lessons