Hey guys I’m back with another post of my series of CXL. So let’s start from where we have left:
I realized that I’m adding too many details and I’m quite lazy to type all of that therefore I’m adding just the overview now.
Mouse tracking: Track people’s mouse movements to see where are they clicking and how far are they scrolling.
Watch the replays of the user sessions to understand where exactly are people facing issues.
Google analytics health checkup: Check the following stuff:
- Profile setup ( configuration and admin )
- Filtering of traffic ( agency, office, data cleaning )
- Goal and funnel configuration ( key reports )
- Code review ( On-page analytics code )
- Page and process instrumentation ( funnel, steps, forms )
- Any issues that would prevent insights.
All of this information is covered in an insanely detailed manner in the course covering what, why, and how exactly to do that.
Let’s move to the next course which is AB testing mastery. This is a highly theoretical course so let’s cover the basics here:
The biggest mistakes in A/B testing are:
- Experimenting on not enough data.
- Not using proper statistics.
- Not knowing if an experiment fell it or not, not really know when to call a winner.
- Not understanding what to experiment on.
When to use A/B testing:
- When you deploy something on your website.
- Can be used for research and it can be split into more parts.
Check abtestguide.com
Power: When you start trying to test on pages with high power (>80%) otherwise you don’t detect effects when there is an effect to be detected. (False negatives)
Significance: When you start trying to test against a high enough significance level (90%) otherwise you’ll declare winners when in reality there isn’t an effect. (False positives)
The 6V conversion canvas includes value, versus, view, voice, validated, and verified.
Create behavioral segments:
Typical e-commerce flow examples:
- All users on your website with enough time to take action.
- All users on your website with at least some interaction.
- All users on your website with heavy interaction.
- All users on your website with clear intent to buy.
- All users on your website that are willing to buy.
- All users on your website that succeed in buying
- All users on your website that return with the intent to buy more.
How to calculate the lengths of your A/B tests:
Use the AB test guide calculator to calculate the lengths of the AB test. Conversion rate can be different in fay or nights, weekday or weekends. Even different times of the day can influence people buying behavior.
AB test should run between 1–4 weeks because if the test runs longer, there is a chance that people will delete that cookie and may land in a different variation of the website which will make all the data useless.
Monitor your AB tests and run experiment dashboards. Stop an AB test when something is broken, If there is an SRM error, or If it’s causing too much loss of money.
AB Test Outcomes:
Before you start analyzing:
- Know the test duration, only check during the test that conversion is coming in and the traffic is evenly distributed.
- Determine how to isolate the test population ( users who have seen the variation )
- Determine the test goals and how to isolate those users ( users who have seen the variation followed by a test goal )
When we start analyzing:
- Analyze in the analytics tool and hot in the test tool.
- Avoid sampling.
- Analyze users, not sessions.
- Analyze users who have converted not users and total conversions.
- Check if the population of users that have seen the test are about the same per variation ( SRM error checks )
Inconclusive result
No winner: It doesn’t mean your hypothesis or design was terrible. We are just unable to prove that it is outperforming the default.
If the result can be implemented: The chances of a negative impact are quite small the variation has a higher measured conversion rate than the default.
Does it add to the button line: No
Should I filter out the users that could not have been influenced: Yes
Should I Digg in all sort of segments to find winners: No
It’s a winner: Implement ASP
Statistics fundamentals for testing:
Sampling: Populations, parameters, and statistics
- The population is the pool of all potential users or people in a group of things that we want to measure.
- What are the examples of population parameters?
- Sample statistics are used to make inferences on the population parameters.
Mean, Variance, and standard deviation:
- Mean is the most common measure of central tendency.
- How spread out the data is what’s known as a common variance.
- The common measure of variability in statistics is the standard deviation.
- The common measure of variability in statistics is the standard deviation.
- Standard deviation is the measure of variance.
- The shape of the spread, or the amount of variance, has a direct impact on the sample size needed to compare the two means.
Confidence Intervals:
Confidence intervals are the range of values, defined in a way that there’s a specific probability of the value of that parameter that lies within.
Confidence interval includes:
- Mean
- Sample
- Variability
- Confidence level
- Confidence intervals are the amount of error allowed in AB testing. It is the measure of the reliability of the estimate.
- True conversion rate can’t be measured.
Statistical significance and the p-value:
- Statistical significance helps us quantify whether a result is likely due to chance.
- The P-value is the probability o obtaining the difference you saw from a sample if there really isn’t a difference for all the users.
- P-value does not tell us that the probability of B>A
- Does not tell us the probability that we will make a mistake in selecting B over A.
Okay, guys that were all for this week. I’m quite behind in my series therefore I will try to add fewer details form now and cover more courses in my blogs.
See you next week.
