Validation Strategies Advanced

Even with JSON mode and Pydantic schemas, AI output needs validation. Structural validity (it is JSON) does not guarantee semantic correctness (the values make sense). This lesson covers building robust validation pipelines for production.

Validation Layers

Validation Pipeline 
# Layer 1: Syntax validation
Can the output be parsed as JSON/XML?

# Layer 2: Schema validation
Does the structure match the expected schema?
Are required fields present? Are types correct?

# Layer 3: Constraint validation
Are values within expected ranges?
Do enums match allowed values?

# Layer 4: Semantic validation
Do the values make logical sense?
Is "age: 500" valid JSON but semantically wrong?

# Layer 5: Business rule validation
Does the output satisfy domain-specific rules?

Retry with Error Feedback

Python 
from pydantic import BaseModel, ValidationError
import json

def parse_with_retry(client, messages, model_cls, max_retries=3):
    """Parse structured output with validation retries."""
    for attempt in range(max_retries):
        response = client.chat.completions.create(
            model="gpt-4o",
            response_format={"type": "json_object"},
            messages=messages
        )
        text = response.choices[0].message.content

        try:
            data = json.loads(text)
            result = model_cls.model_validate(data)
            return result
        except (json.JSONDecodeError, ValidationError) as e:
            # Feed the error back to the model
            messages.append({
                "role": "assistant",
                "content": text
            })
            messages.append({
                "role": "user",
                "content": f"Validation error: {str(e)}. Please fix and try again."
            })

    raise ValueError(f"Failed to get valid output after {max_retries} attempts")

Fallback Parsing

Python 
import json
import re

def robust_json_parse(text):
    """Try multiple strategies to extract JSON from model output."""

    # Strategy 1: Direct parse
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass

    # Strategy 2: Extract from code block
    match = re.search(r'```(?:json)?\s*([\s\S]*?)\s*```', text)
    if match:
        try:
            return json.loads(match.group(1))
        except json.JSONDecodeError:
            pass

    # Strategy 3: Find first { to last }
    start = text.find('{')
    end = text.rfind('}')
    if start != -1 and end != -1:
        try:
            return json.loads(text[start:end + 1])
        except json.JSONDecodeError:
            pass

    raise ValueError("Could not extract valid JSON from response")

Graceful Degradation

Production Pattern: When validation fails, do not just throw an error. Degrade gracefully:
  1. Retry with error feedback (up to 3 times)
  2. Try fallback parsing strategies
  3. Use partial results with defaults for missing fields
  4. Fall back to a simpler model or prompt
  5. Queue for human review if all automated approaches fail

Monitoring Parse Failures

Python 
class ParseMetrics:
    def __init__(self):
        self.total = 0
        self.success_first_try = 0
        self.success_with_retry = 0
        self.failures = 0
        self.failure_types = {}

    def record(self, success, attempt, error_type=None):
        self.total += 1
        if success and attempt == 1:
            self.success_first_try += 1
        elif success:
            self.success_with_retry += 1
        else:
            self.failures += 1
            if error_type:
                self.failure_types[error_type] = self.failure_types.get(error_type, 0) + 1

    @property
    def success_rate(self):
        return (self.success_first_try + self.success_with_retry) / max(self.total, 1)
Cost Awareness: Every retry costs additional API tokens. Track retry rates and costs separately. If a particular schema consistently requires retries, improve the prompt or simplify the schema rather than relying on retries.
← XML Output Best Practices →