There are 2 stages to testing analysis.
Stage 1 = Quantitative analysis
Stage 2 = Qualitative analysis
Stage 1 is higher priority and defines the test results.
That is, from the variation/s tested, did you meet the required uplift during the time defined? Was there a winning variation? Did multiple variations have statistical significance?
Once we’ve identified which variation/s had the greatest impact, we can move on to stage 2 to better understand why they performed better & why the losing variation/s performed worse.
This can be done via HotJar session recordings, heatmaps, clickmaps, etc.
The two concluding questions to every test are the following:
Did we achieve the desired impact from this test?
What else did we learn about our customers from this test?
IF we’ve achieved the desired impact, we can implement the winning variation site-wide & continue optimizing further.
IF we have not achieved the desired impact, we want to document the changes that we’ve made, and keep this information in a handy place to avoid performing the same experiment in the future.