Who can use this feature?
- Available with Enterprise and Advanced plans.
- Explorers can view and create. Admins, architects, and standard users can view, create, and save.
This is part 5/5 of a series explaining Conversions:
- Introduction to Conversions
- Your first Conversion Analysis
- An overview of your Conversions results
- Digging into & sharing your Conversions results
- Digging into your Conversions results II
In the previous article, we explained how to dig deeper into the details of an opportunity. In this article, we're providing a technical addendum that provides full transparency into the calculations behind the Conversions magic. It’s especially relevant to data scientists, analysts, and the technically inclined.
Three ways we find Conversion impacts for you
In order to maximize insights with Conversions, it’s important to understand the special cases that arise when we automatically compare the performance of an “affected” and “unaffected” group through a funnel, for a given signal. Behind the scenes, we actually compute different versions of the impact formula and show you only the most relevant one that applies.
Note: When interpreting your funnel conversion rates between affected and unaffected groups, be aware that the conversion impact of an experience may not occur at exactly the same step of the funnel as the experience itself. There may be delayed effects especially if experiences such as javascript errors can accumulate or cause downstream knock-on effects for the user.
Here are the 3 approaches that we take to detect a significant conversion impact for a given funnel. Conversions will first attempt to find a significant impact from the first approach (“end to end”) and will present that impact if it succeeds. Otherwise, it will try the second approach, and so on. Conversions will show the result from the first approach that succeeds.
1. End to End of the Funnel
In the simplest case, your affected and unaffected groups perform like two textbook funnels: every step shows some dropoff, and the end-to-end conversion rate of the affected group is low enough compared to the unaffected group to be considered statistically significant.
2. Mid to End of the Funnel
If the pink dotted box in the Analysis View is only around certain steps in the funnel, while the others are grayed out, this means we've detected that a given signal doesn't occur until mid-way through the funnel.
We do this to ensure that impacts from that signal can be brought to your attention. If we only calculated the impact end-to-end, we might miss significant differences in conversion rate that only apply after the event has happened.
Note: Normally, the first highlighted step is the step on which the signal occurs. Whether you can be sure of this for your funnel depends on what type of signal you are analyzing and how much you know about its origin.
3. Step to Step within the Funnel
There are times when a more granular approach is needed to uncover useful insights. Even if you’ve thought about step-to-step transition rates before (which are sometimes known as micro-conversions), comparing these types of transitions between affected and unaffected groups is complex to interpret and may be unfamiliar to you.
Let’s say your desktop users typically convert more frequently than mobile users, but that there’s an impactful bug on the desktop version of your website that affects everyone in that cohort and reduces their chance of converting. Suppose the bug doesn’t impact desktop conversion enough to make it overall below that of the mobile group, otherwise Conversions could surface the issue in one of the previous approaches.
From an end-to-end viewpoint over all your users, the affected group (in this case, Desktop users) is still converting better overall than the unaffected group (Mobile users), so there would be nothing to report using the previous two approaches. But if there’s a significant loss of customers between two steps for Desktop users compared to the loss on the same steps for Mobile users, then Conversions might tell you that you could be converting even more desktop users if you addressed this bug.
Note: We can’t guarantee that Conversions will catch every possible impact in a stepwise approach because it depends on the exact pattern of micro-conversion rates in the unaffected group’s funnel.
The stepwise type of opportunity is not easy to detect with tools that only analyze one cohort’s funnel activity at a time. Conversions always analyzes two related cohorts at the same time, which makes this comparison possible.
In the unlikely situation that we find multiple stepwise impacts for one signal in the same funnel analysis, we only surface the one with the highest lost conversions.