A/B Testing
Media A/B Testing for Ad Headline Optimization
Client
Nike, Peloton, Critical Mass
YeAr
2022 - 2025
Category
Experimentation & Causal Inference
Service
Increasing Revenue / ROI
Tools / Languages Used
- Meta Ads Manager & Google Ads for experiment setup
- Python (Pandas, SciPy, Statsmodels) for statistical analysis
- SQL for campaign performance extraction
- Tableau for visualization and stakeholder reporting
Technical Skills
- Digital experiment design and randomization
- Statistical testing (two-proportion z-test)
- ROAS, CTR, CPC
- ROI reporting and data visualization
Soft Skills
- Collaboration with marketing and creative teams
- Translating behavioral insights into messaging strategy
- Communicating results to non-technical stakeholders
- Building an experimentation mindset within the team
Step 1: Exploratory Data ANalysis
- Analyzed historical campaign data to identify variability in performance across creative headlines.
- Found that ads using emotive or benefit-led copy (“Save time every day”) had higher CTR but inconsistent conversions.
- Determined a need to validate headline messaging effectiveness with controlled testing.
Step 2: Solution Design
- Defined hypothesis: Headline emphasizing customer benefit will outperform feature-based headline in CTR and conversions.
- Created two ad variants with identical visuals, targeting, and spend, differing only by headline text in the creative.
- Primary KPI: Return on ad spend (ROAS)
- Secondary KPIs: Click-through rate (CTR), and cost per click (CPC)
- Ran the test for 3 weeks across Facebook and Instagram placements with even audience distribution.
Step 3: Model Assessment
- After 3 weeks of equal spend:
- Variant A (Feature-Focused): CTR 1.82%, CVR 3.1%
- Variant B (Benefit-Focused): CTR 2.14%, CVR 3.6%
- p-value = 0.012 → statistically significant improvement in CTR
- Conversion rate uplift was directionally positive but not statistically significant (p = 0.09).
- Post-hoc analysis showed stronger response among mobile users and younger demographics.
Step 4: Results / How It’s Used
- Benefit-focused headline drove ~18% higher CTR and improved engagement quality.
- Insights informed copywriting guidelines for future ad campaigns — focusing on emotional and outcome-driven messaging.
- Marketing team reallocated 40% of creative testing budget toward headline experimentation.
- The test framework was expanded to include CTA text and value proposition wording in upcoming campaigns.
- Benefit-focused headline drove ~18% higher CTR and improved engagement quality.
- Insights informed copywriting guidelines for future ad campaigns — focusing on emotional and outcome-driven messaging.
- Marketing team reallocated 40% of creative testing budget toward headline experimentation.
- The test framework was expanded to include CTA text and value proposition wording in upcoming campaigns.
