Lilly Tech Systems
AI Audit & Assurance
Master AI audit and assurance. 50 deep dives across 300 lessons covering foundations (audit purpose, AI vs IT audit, audit lifecycle, standards landscape, auditor competencies), audit frameworks & standards (ISACA, IIA Three Lines, ISO 42001 audit, COBIT, SOC 2, AICPA, EU AI Act conformity, IAASB ISAE 3000), internal audit programs (annual plan, scoping, charter, function design, methodology, reporting), external & third-party audits (independent, certification, regulator, assurance, attestation), audit planning & scoping (planning, materiality, sampling, evidence requests, charter), technical AI audits (bias, fairness, explainability, robustness, security, privacy, model card, dataset), process & governance audits (governance, lifecycle, MLOps, vendor, incident response, training), audit tools & techniques (automated tools, evidence repos, workpapers, analytics), and reporting & remediation (report writing, findings tracking).
AI Audit & Assurance is the track for the people who verify that AI systems actually do what their owners claim. That population is growing quickly: internal auditors expanding into AI, external auditors being retained for AI-specific engagements, certification bodies now conducting ISO/IEC 42001 audits, regulators running market surveillance, and enterprise customers demanding attestation reports before they will buy. The discipline has its own methods, which this track covers.
The lessons are grounded in the audit standards that governing bodies have already adopted (ISACA's AI audit toolkit, IIA three lines for AI, AICPA's AI ethics and trust work, IAASB's ISAE 3000, the EU AI Act conformity assessment regime). Technical audits (bias, fairness, explainability, robustness, security, privacy, model card, dataset) get the same depth as process and governance audits. The goal is that a reader of the track can plan, run, and report an AI audit that holds up to independent review.
All Topics
50 AI audit and assurance topics organized into 9 categories. Each has 6 detailed lessons with frameworks, audit programs, and operational templates.
Foundations
AI Audit Foundations
Master the foundations of AI audit. Learn the audit purpose, stakeholders, audit types (internal, external, certification, regulator), and what makes AI audit different from traditional IT audit.
6 LessonsAI Audit vs Traditional IT Audit
Understand how AI audit differs from traditional IT audit. Learn the new audit objectives (fairness, explainability, robustness), evidence types, sampling challenges, and skill gap.
6 LessonsAI Audit Lifecycle
Walk through the AI audit lifecycle. Learn planning, fieldwork, testing, evaluation, reporting, and follow-up - and how each stage adapts for AI systems.
6 LessonsAI Audit Standards Landscape
Map the AI audit standards landscape. Learn the major bodies (ISACA, IIA, AICPA, IAASB, ISO/IEC, NIST), how they relate, and how to choose the right one for an engagement.
6 LessonsAI Auditor Competencies
Build AI auditor competencies. Learn the technical, regulatory, governance, and ethical knowledge auditors need; certifications (CISA, CRISC, AIGP, ISO 42001 lead auditor); and continuous learning.
6 LessonsAudit Quality Control
Establish audit quality control. Learn engagement quality review, peer review, hot/cold reviews, documentation standards, and continuous improvement.
6 LessonsAudit Frameworks & Standards
ISACA AI Audit Framework
Master the ISACA AI Audit Toolkit and Framework. Learn the audit objectives, control categories, testing procedures, and the ISACA digital trust model.
6 LessonsIIA Three Lines Model for AI
Apply the IIA Three Lines Model to AI. Learn first-line product/eng accountability, second-line risk/compliance oversight, third-line internal audit assurance, and board governance.
6 LessonsISO/IEC 42001 Audit
Audit against ISO/IEC 42001. Learn AIMS clause-by-clause audit, Annex A control testing, certification audit (Stage 1 & 2), surveillance, and recertification.
6 LessonsCOBIT 2019 for AI
Apply COBIT 2019 to AI governance audit. Learn the governance and management objectives, design factors for AI, and the COBIT capability assessment.
6 LessonsSOC 2 for AI Services
Apply SOC 2 to AI services. Learn how Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy) extend to AI systems and emerging AI-specific criteria.
6 LessonsAICPA AI Assurance
Use AICPA AI assurance offerings. Learn the AICPA AI ethics framework, system trust model for AI, and the emerging AI assurance attestation guidance.
6 LessonsEU AI Act Conformity Assessment
Conduct EU AI Act conformity assessment. Learn the conformity-assessment options for high-risk AI, notified body process, internal control type, harmonised standards, and CE marking.
6 LessonsIAASB ISAE 3000 for AI
Apply IAASB ISAE 3000 (Revised) to AI assurance. Learn reasonable vs limited assurance, suitable criteria, evidence requirements, and report formats for AI engagements.
6 LessonsInternal Audit Programs
Annual AI Audit Plan
Build an annual AI audit plan. Learn risk-based prioritisation, audit universe definition, resource planning, and aligning with the enterprise risk appetite.
6 LessonsRisk-Based Audit Scoping
Apply risk-based scoping for AI audits. Learn how to translate risk register entries into audit objectives, scope statements, and test procedures.
6 LessonsAudit Charter & Mandate
Draft the AI audit charter and mandate. Learn the IIA charter requirements, board approval, independence safeguards, and how to extend the mandate to AI.
6 LessonsInternal Audit Function Design for AI
Design the IA function for AI. Learn org placement, roles & responsibilities, sourcing model (in-house, co-source, outsource), and the AI center-of-excellence pattern.
6 LessonsAudit Methodology Selection
Select the right audit methodology. Learn risk-control matrix, process audits, attribute sampling, agile audit, continuous audit, and choosing per engagement.
6 LessonsReporting Cadence
Establish IA reporting cadence. Learn engagement reports, periodic audit committee reports, KPI dashboards, and integration with risk reporting.
6 LessonsExternal & Third-Party Audits
Independent External Audit
Engage an independent external auditor. Learn auditor selection, engagement letters, fee models, scope negotiation, and managing the audit relationship.
6 LessonsCertification Audits
Run certification audits. Learn certification body selection, accredited bodies, ISO 42001 certification path, surveillance schedule, and certificate maintenance.
6 LessonsRegulator Audits & Investigations
Prepare for regulator audits and investigations. Learn the FTC/EEOC/CNIL/ICO playbook, the EU AI Office market surveillance, document holds, and litigation-ready evidence.
6 LessonsAssurance Engagements
Scope and run AI assurance engagements. Learn assurance vs audit, attestation vs direct, agreed-upon procedures, and choosing assurance level (reasonable vs limited).
6 LessonsAttestation Reports for AI
Issue and consume AI attestation reports. Learn SOC 2/3 for AI, ISO 42001 certificates, AICPA AI attestation templates, and how to interpret a report you receive.
6 LessonsAudit Planning & Scoping
Audit Planning Deep Dive
Master audit planning for AI. Learn risk assessment, audit scope, audit program design, kickoff meetings, stakeholder mapping, and timeline.
6 LessonsMateriality for AI
Apply materiality concepts to AI audits. Learn quantitative materiality (financial, scale-based), qualitative materiality (harm-based), and combining the two.
6 LessonsSampling Strategy for AI
Design sampling strategies for AI audits. Learn statistical vs judgmental, attribute vs variables, sample size calculation, slicing for fairness, and corner-case sampling.
6 LessonsEvidence Requests (PBC Lists)
Design evidence requests for AI audits. Learn the PBC (Prepared by Client) list structure, AI-specific evidence types, secure delivery, and tracking.
6 LessonsEngagement Charter & Memo
Draft engagement charter and memo. Learn objectives, scope, criteria, methodology, timeline, deliverables, and management acknowledgement.
6 LessonsTechnical AI Audits
Bias Audit
Conduct a bias audit. Learn protected-class definition, fairness metrics testing, sub-group analysis, NYC AEDT-style audit, CO AI Act assessment, and reporting.
6 LessonsFairness Audit
Conduct a fairness audit beyond bias metrics. Learn procedural fairness (process), distributive fairness (outcomes), counterfactual fairness, and intersectional analysis.
6 LessonsExplainability Audit
Audit AI explainability. Learn local vs global explanations, LIME/SHAP/anchors testing, faithfulness checks, audience-appropriate explanations, and EU AI Act Article 13.
6 LessonsRobustness Audit
Audit AI robustness. Learn input perturbation testing, OOD evaluation, adversarial testing, stress testing, and degradation monitoring.
6 LessonsAI Security Audit
Audit AI security. Learn ATLAS technique coverage, prompt-injection testing, model-extraction defense, supply-chain integrity (AIBOM), and red-team exercises.
6 LessonsAI Privacy Audit
Audit AI privacy. Learn DPIA review, training-data privacy, PII-in-output testing, membership-inference resistance, DP/FL/SMC verification, and DSAR handling.
6 LessonsModel Card Audit
Audit model cards. Learn the Mitchell et al. template, completeness checks, accuracy verification, intended-use vs actual-use audit, and disclosure adequacy.
6 LessonsDataset Audit
Audit training datasets. Learn datasheets for datasets, lineage and provenance, license review, copyright posture, opt-out compliance, and labeling-process audit.
6 LessonsProcess & Governance Audits
AI Governance Audit
Audit AI governance. Learn governance body effectiveness, policy/standard adequacy, decision-rights clarity, and operating-model maturity assessment.
6 LessonsAI Lifecycle Audit
Audit the AI lifecycle. Learn stage-gate audit, lineage and reproducibility verification, model approval audit, and retirement / decommissioning audit.
6 LessonsMLOps Audit
Audit MLOps. Learn CI/CD pipeline audit, feature store integrity, model registry control, deployment approvals, and rollback testing.
6 LessonsVendor AI Audit
Audit AI vendors. Learn vendor due diligence, on-site/remote vendor audit, attestation report consumption, ongoing monitoring, and concentration risk.
6 LessonsIncident Response Audit
Audit AI incident response. Learn runbook adequacy, exercise/tabletop review, incident-log audit, regulator-notification compliance, and post-incident review effectiveness.
6 LessonsTraining & Competency Audit
Audit training and competency. Learn role-based training matrix, completion tracking, effectiveness assessment, EU AI Act Article 4 (AI literacy), and refresher cadence.
6 LessonsAudit Tools & Techniques
Automated Audit Tools
Use automated tools for AI audit. Learn fairness toolkits (AIF360, Fairlearn), explainability tools (SHAP, LIME, Captum), evaluation harnesses (lm-eval-harness), and orchestration.
6 LessonsEvidence Repositories
Build evidence repositories. Learn repository design (immutable storage, versioning, retention), access controls, audit trail, and integration with GRC platforms.
6 LessonsAudit Workpapers
Create audit workpapers. Learn workpaper standards (PCAOB, IIA), templates, indexing & cross-referencing, electronic workpaper systems, and review trail.
6 LessonsAudit Analytics for AI
Apply audit analytics to AI. Learn population testing (vs sampling), anomaly detection, exception reporting, continuous auditing, and visualization for findings.
6 LessonsReporting & Remediation
Audit Report Writing
Write effective AI audit reports. Learn structure (executive summary, findings, recommendations, mgmt response), severity rating, evidence citation, and stakeholder-tailored versions.
6 LessonsFindings & Remediation Tracking
Track findings and remediation. Learn finding workflow, owner assignment, due-date discipline, validation testing, escalation, and closing the loop with evidence.
6 Lessons