Increasing Live stream engagement with A/B testing and research
Platform:
Mobile apps
Year:
2022
Tools:
Figma, Userzoom, Firebase, Adobe Analytics
My responsibilities:
Quan research (200 responses)
A/B tests planning
Data analysis
Low & Hi Wireframes
Prototypes
Usability testing & validation report
The challenge
With research and surveys showing that more than half of all video content is now consumed on mobile devices such as smartphones and tablets, Sky News has started to offer to its users a mobile-friendly 24x7 live streaming in vertical 9:16 format, as part of the wider digital transformation strategy (5% video consumption).
However, two 4 week A/B tests were launched in February 2022 showed that engagement for Live TV in vertical was falling behind the horizontal format.
Baseline - Horizontal stream
Variant - Horizontal stream
What did we already know?
Behavioural insights
- The typical peak hours for Live TV on mobile are 7am-9am and 5am-7pm (UK time)
- App engagement decreased since last December “Omicron effect.” Then, it started to increase steadily because of the “Russian invasion of Ukraine”
- Mobile and tablet receive a considerable amount of traffic from social media platforms
Data insights
- 28% of users visited app watched Live TV
- Avg time spent on Live TV (baseline) < 8s for iOS and < 6s for Android
Objectives
- Find out why users engaged less with the Vertical stream
- Usability testing for suggested solutions
- A/B test with new variants
Understand user needs
Method
The quantitive research showed that 2/3 of users expected to play Live TV full screen. Therefore, the ideal solution to have a full-screen button and let users rotate their screen to match their orientation preferences
Key learnings
- Order of destination preferences for mobile: Youtube > Tiktok > Sky News
- Only tested on live stream which is long-form whereas vertical video shines in short-form
- Two common behaviours when playing prototypes on either vertical or horizontal stream: hiding other parts of the screen (cinema mode) and rotating the screen -> Users just simply wanted to get a full screen player
- People generally preferred vertical format but would want to be able to choose the stream format
- The ideal format of the video is determined by other factors, like content and length of video
Youtube
TikTok
Sky news
Facbook
Different platforms where users catched up Sky News Live TV
Insights: Users subscribed / followed Sky News on social media so they would get the Live TV on their feed and notification for Breaking News
Setting direction
The ideal solution
The quantitive research showed that 2/3 of users expected to play Live TV full screen. Therefore, the ideal solution to have a full-screen button and let users rotate their screen to match their orientation preferences
However, it wasn't possible due to these restrains
- We are dealing with a 15-25 second delay in vertical stream (time for AI to convert the screen estate)
- Risk of missing important moments while switching streams
- Users aren’t aware of 2 different streams and expect a seamless transition
- Research says many users disable “auto-rotation”
Distilled thinking into a hypothesis
The Live TV vertical stream doesn't provide full screen experience all the time. Improving the quality isn’t something we can solve right now. However, giving users the viewing options, with a simple and non-intrusive to switch can improve engagement as users now can choose their preferred format.
Usability testing & iterations
Some of the experimental options
The quantitive research showed that 2/3 of users expected to play Live TV full screen. Therefore, the ideal solution to have a full-screen button and let users rotate their screen to match their orientation preferences
Consider factors
- Barrier to stream
- Easy to switch
- Clear explainer and instruction
- Switch button discoverability
Refined solution
Key takeaways
- Most users noticed the toggle and naturally interacted with it. They kept switching it on and off to see what that would give them
- The toggle was easier to access, more visible and did not disappear with the player controls
- All users said the delay between feeds was not an issue to them
- All users said they would like to have both streams and the option to switch. Though, more users preferred to have horizontal stream as default
- Some users perceived the vertical stream as full screen mode while holding their phone upright. This led them to believe that because this was full screen, they would be able to rotate their phones and go into full landscape.
Expected behaviour flow
Second A/B testing result
Previous test
Second test
- After 2 week testing, the result showed an overall of 2 mins increase across all devices and streams.
- Over 80% of users landing on Live TV page used the toggle to switch.
- Despite the fact that more users from the A/B test still switched back to the Horizontal stream, there were several positive feedback responses when asked.
User feedback
Conclusion
From user testing, vertical video seems to be a great option for a couple of reasons: it provides a full screen option for those who like to keep their phone upright when watching videos, and the zoomed in view allows users to see details better and read the text on screen a little bit easier, as the font on the original stream is very small.
The research showed that some users find value and delight in the option to toggle the vertical stream on if they like.
However, the test made it clear that when the vertical stream showed horizontal content (over 60% of the time), the value of the feature decreased significantly. For this reason, our recommendation is to only implement the toggle once the vertical stream shows more full screen content. Doing so will increase the likelihood of users’ delight and subsequent continuous engagement with the feature.
Therefore, we recommended to add more live channels as users mentioned about being able to simply swipe up to change channel or a simple way to access short form videos in vertical format.