Lilly Tech Systems
Intermediate
AI-Powered A/B Testing
AI's ability to generate dozens of copy variants makes systematic email testing faster and more thorough than ever. Learn to test subject lines, body copy, CTAs, and send times to continuously improve your email performance.
What to Test in Emails
| Element | Variables | Primary Metric |
|---|---|---|
| Subject Line | Length, tone, personalization, emoji, formula | Open rate |
| Preview Text | Complement vs. continuation, length, CTA | Open rate |
| Body Copy | Length, tone, structure, storytelling vs. direct | Click-through rate |
| CTA | Button text, color, placement, number of CTAs | Click-through rate |
| Send Time | Day of week, time of day, timezone | Open rate + engagement |
| From Name | Company vs. person vs. combo | Open rate + trust |
Pro Tip: Test one variable at a time for clean results. AI can generate variant pairs that differ in only one dimension, making it easy to isolate what drives performance differences.
AI Testing Workflow
- Hypothesis: Define what you expect to perform better and why
- Generate variants: Use AI to create test versions that differ in the target variable only
- Set sample size: Ensure each variant reaches enough subscribers for statistical significance
- Run test: Send to a subset, measure for 2-4 hours (or as appropriate)
- Analyze: Check statistical significance before declaring a winner
- Apply: Send the winning version to remaining subscribers and document learnings
Advanced Testing Strategies
Multivariate Testing
Test multiple elements simultaneously. AI generates all combinations of subject lines, body copy, and CTAs for factorial analysis.
Sequential Testing
Build on previous test winners. Each round's champion becomes the control for the next round of AI-generated challengers.
Segment-Level Testing
Different segments may respond differently. AI generates segment-specific variants and tests them against each audience group.
Multi-Armed Bandit
AI automatically shifts traffic to the best-performing variant in real time, maximizing results while still gathering data.
Building a Testing Culture
- Test every email: Never send without at least a subject line test — AI makes variant creation effortless
- Document learnings: Build a knowledge base of what works for your audience
- Share results: Testing insights from email apply to landing pages, ads, and social content too
- Avoid false positives: Wait for statistical significance (95% confidence) before drawing conclusions