Intermediate

AI Risk Assessment

Risk assessment is the systematic process of identifying, analyzing, and evaluating AI risks. It forms the foundation for all risk management decisions.

AI Risk Taxonomy

A risk taxonomy provides a structured classification of potential AI risks. Using a standardized taxonomy ensures comprehensive coverage and consistent communication across teams:

Risk Domain Risk Categories Assessment Method
Technical Model accuracy, robustness, drift, failure modes Quantitative testing and monitoring
Ethical Bias, fairness, discrimination, human autonomy Bias audits, fairness metrics, stakeholder reviews
Legal Regulatory compliance, liability, intellectual property Legal review, compliance mapping
Operational System availability, scalability, dependency risks Operational testing, resilience assessment
Societal Environmental impact, labor displacement, power concentration Impact assessments, community engagement

Likelihood and Impact Analysis

  1. Define Likelihood Scales

    Establish clear criteria for likelihood levels. For AI systems, consider: How often does the model encounter edge cases? What is the probability of adversarial attacks? How likely is concept drift given data volatility?

  2. Define Impact Scales

    Categorize impact across dimensions: individual harm (physical, financial, psychological), organizational harm (regulatory, reputational, financial), and societal harm (discrimination, trust erosion, democratic processes).

  3. Build a Risk Matrix

    Combine likelihood and impact into a risk matrix. Score each identified risk and plot it on the matrix. This visual representation helps prioritize risks and communicate them to non-technical stakeholders.

  4. Consider Cascading Effects

    AI risks can cascade. A model accuracy degradation may lead to user harm, which triggers regulatory action, causing reputational damage. Map these chain reactions to understand true risk exposure.

Stakeholder Impact Mapping

⚠
Critical Step: Many AI risk assessments focus only on technical risks. The most consequential risks often involve harm to affected individuals and communities who have no direct relationship with the organization deploying the AI system. Stakeholder mapping must include these indirect stakeholders.

Direct Users

People who interact directly with the AI system. Assess risks related to usability, trust, over-reliance, and decision support quality. Consider how errors affect their work and decisions.

Affected Individuals

People who are subject to AI-driven decisions but may not use the system themselves. Assess risks of bias, discrimination, denial of services, and lack of recourse or appeal mechanisms.

Deploying Organization

The entity operating the AI system. Assess regulatory risk, liability exposure, reputational risk, operational dependency, and the cost of risk mitigation versus potential harm.

Broader Society

Communities and societal systems affected by widespread AI deployment. Assess systemic risks including labor market impacts, environmental costs of training, and effects on democratic processes.

Risk Scoring Methodologies

  • Qualitative scoring: Use expert judgment with structured rubrics. Best for novel AI risks where historical data is unavailable. Use Delphi methods to aggregate expert opinions.
  • Semi-quantitative scoring: Assign numerical scores to qualitative categories (e.g., 1-5 for likelihood and impact). Calculate composite risk scores. Most common approach for AI risk assessment.
  • Quantitative scoring: Use statistical methods, historical incident data, and simulation to estimate risk probability and expected loss. Requires mature data collection and risk modeling capabilities.
  • Scenario-based assessment: Develop detailed failure scenarios and assess their consequences. Particularly useful for high-stakes AI systems where worst-case analysis is critical.
💡
Next Up: In the next lesson, we explore risk mitigation strategies — technical controls, organizational controls, risk transfer, and residual risk management.