Stories · Data
A beautiful test result
You run growth. Last week's A/B test of a new ad channel, X, showed an 18% higher conversion rate than the control group. Your boss asks whether to move half of next quarter's $600k budget to channel X. You go to check the details with your analyst, Leo.
Who you're talking to
Leo, data analyst.
(pushes up his glasses) You've seen the report? What would you like to know?
Questions to start with
- “How long did the test run?”
- “Which users did the 18% come from?”
- “How confident are you in this result?”
Leo won't volunteer everything. Some of what matters only comes out if you ask the right follow-up questions.
Questioning habits this story trains
- Look for another explanation. For example: “Could something else have caused it?”
- Ask about the downside they skipped. For example: “What's the worst that could happen?”
- Get the story behind a number. For example: “From what to what? Compared with what?”
- Ask where the evidence comes from. For example: “How many cases like that? Who says so?”
The decision
When you've heard enough, you give Leo your answer face to face. The choices include:
- Move $300k to channel X next quarter
- Spend $60k on a four-week test that only looks at new users
- Skip channel X and keep things as they are
- Negotiate the price first, then scale up in stages based on results
Then you read what happened a year later, see how sound your decision was, and get notes on the key moments of your conversation with a better way to say each.
Rehearse this conversation for free. New accounts get 2 free rounds. One round takes under 10 minutes.
More stories
- The tempting offer · Career
- Ship on time or delay · Management
- A second café? · Startup
- A friend asks you to invest · Investing
- Two candidates, one job · Management
- Should Dad move in? · Family
- Quit for a full-time MBA · Growth