Description / Hypothesis
Adding a video testimonial to the sell timeshare page will increase user engagement with the page and encourage more to submit a lead. This new version will test the video in a slide in on desktop and immediately below the form on mobile, with the goal of increasing play rate from previous versions and driving CVR.
Post-Mortem Analysis
Key Learnings
There was a shift in performance with this experiment between mobile and desktop devices, with mobile users receiving the video asset more favorably than desktop users.
CVR (all Owner goals) Mobile:
Control: 5.62%
Variant: 6.29%
% Lift: +11.92%
Confidence: 86%
Stat sig: NO
CVR (all Owner goals) Desktop:
Control: 9.92%
Variant: 8.17%
% Lift: -17.64%
Confidence: 98%
Stat sig: YES
Other Observations
We did manage to increase the play rate for both mobile and desktop iterations of this experiment, with the desktop video finishing at 5.61% and the mobile video at 10.13%. Those increases are statistically significant from the previous test, which had a play rate of 2.8% (combined).
It is possible that desktop users became discouraged by the slide in feature. It can be a knee-jerk reaction to immediately close something as soon as it pops into view, especially if they are in the middle of digesting the information in the form or in the hero.
Of users who played the video, the CVR for the mobile segment was markedly higher than described above, 6.79%. This could mean that users who watch the video become more qualified to submit a lead than those who don’t (however, this result was not stat sig).
Decisions to Implement
None. As long as we have these video testimonial assets, they will be a continuous part of our optimization strategy and testing backlog.