Implementation worksheet · 5 min read
A Product Tour Completion Versus Activation Report
Report four cohorts rather than one completion percentage: users who completed the tour, users who started and dropped, users who were eligible and never started, and users who were never shown it. Then compare activation rate across all four. Completion rate on its own is a vanity metric because completers are self-selected — they were more motivated before the tour began, so they would have activated at a higher rate regardless. The honest comparison is completers against eligible non-completers, and the only way to remove selection bias entirely is a holdout that is never shown the tour at all.
Almost every onboarding report leads with completion rate, and it always looks good: the people who finish a tour are the people who were already engaged. The report below is what tells you whether the tour did anything.
Put it into practice
1. Define eligibility before exposure
Who should have seen it — by role, plan, or lifecycle stage. Reporting on 'users shown the tour' hides the ones your targeting missed and inflates every downstream rate.
2. Split into the four cohorts
Completed, started-and-dropped, eligible-never-started, and a holdout never shown. The third cohort is usually the largest and is routinely missing from tour reports.
3. Compare activation, not completion
Activation rate per cohort, using your existing activation definition. If completers and eligible non-completers activate at similar rates, the tour is selecting motivated users rather than creating them.
4. Keep a small never-shown holdout
5-10% of eligible users, permanently. It is the only cohort that removes self-selection, and it costs very little once the tour is running.
5. Report drop-off by step alongside it
Where people leave tells you what to cut. Step-two abandonment usually means the tour is too long, not that the content is wrong.
Tour effectiveness report
Copy this structure into your review document and record your observed result for each row.
| Cohort | Users | Activation rate | Difference vs holdout |
|---|---|---|---|
| Completed the tour | |||
| Started, dropped | |||
| Eligible, never started | |||
| Holdout (never shown) | — |
A failure worth checking
The selection-bias win: a tour reports 72% completion and completers activate at three times the rate of everyone else, so the team builds four more tours. The never-shown holdout would have shown that motivated users activate at nearly that rate anyway — the tour was a filter, not a cause, and the four new tours add prompts without adding activation.
Common questions
Is completion rate useless?
Not useless — it is a diagnostic for tour length and clarity, and step-level drop-off is genuinely actionable. It is just not evidence of effectiveness, which is how it is usually presented.
What if we cannot run a holdout?
Then compare completers with eligible non-completers and state the selection-bias caveat explicitly in the report. That is weaker evidence, honestly labelled, which is better than a strong claim you cannot support.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.