Background
During my time at Booking.com, I worked for several years on the reviews team and I was the designer in charge of the reviews collection flow. After a user completed a trip, we asked them for reviews of hotels, cities, and points of interest they visited. This information was then used to inform other travelers and help them to have the best possible trip.
This experiment focused on users who had recently stayed at a hotel but had not yet written their hotel review – specifically users who were visiting the site on a mobile device.
Hypothesis
Based on previous experiments, we knew that if a user was visiting Booking.com within a few weeks of a recent hotel stay, they were most likely there to write a hotel review and not looking to book another hotel. Our average users would book a hotel stay roughly every 4 months so intuitively it makes sense that they aren’t looking to book a hotel immediately after staying in one.
This realization meant that we could potentially prioritize “write a review” actions over some other “book a hotel” actions without losing hotel booking revenue.
This experiment was focused on the mobile version of the Booking.com homepage where there was a “write a review” CTA placed below some “book a hotel” CTAs. Our idea was simple enough: let’s move the “write a review” CTA up above some of the other content on the page. This experiment only applied to users who still had hotel reviews to write and users without reviews to write were excluded.
Results
We ran the experiment for four weeks, with a total of 179k visitors in both variants. Our primary goal review submission increased 3.86% in the “B” version of the test. Additionally, we tracked the net conversion metric to be sure that we were not increasing reviews at the cost of hotel bookings. The data indicated no significant change in that metric so we felt safe to implement the change.
Learnings
We learned a few things here:
- Moving things up the page will result in more attention paid.
- Not all visitors on Booking.com are there to book a hotel.
- Personalizing the content we show certain user segments can yield better results for those users and our KPIs.