Background
During my time at Booking.com, I worked for several years on the reviews team and I was the designer in charge of the reviews collection flow. After a user completed a trip, we asked them for reviews of hotels, cities, and points of interest they visited. This information was then used to inform other travelers and help them to have the best possible trip.
This experiment focused on users who had recently stayed at a hotel but had not yet written their hotel review – specifically users who were visiting the hotel page on a mobile device.
Hypothesis
Based on previous experiments we knew that if a user was visiting Booking.com within a few weeks of a recent hotel stay, they were most likely there to write a hotel review and not looking to book another hotel. Our average users would book a hotel stay roughly every 4 months so intuitively it makes sense that they aren’t looking to book a hotel immediately after staying in one.
This realization meant that we could emphasize “write a review” actions for those users who had recently stayed in a hotel.
Additionally, other experiments had shown us that using a smiley rating to rate hotels helped to engage users and made them more likely to rate and then review a hotel.
Combining these two key pieces of data we decided to improve the existing review CTA on the hotel page by giving the user the option to rate the hotel with smileys. Adding the smileys to the existing CTA block also increased the size of the block overall, helping to improve its visibility.
We tracked hotel review submission as the primary metric but also tracked our overall net conversion metric to make sure that we were not decreasing hotel bookings in this experiment.
Results
We ran the experiment for four weeks, with a total of 125k visitors in both variants. Our primary goal review submissions increased 18.31% (± 2.39%) in the “B” version of the test. Additionally, we tracked the net conversions metric to be sure that we were not increasing reviews at the cost of hotel bookings. The data indicated no significant change in that metric so we felt safe to implement the change.
Learnings
This is a great example of how we can use insights from our past experiments to inspire future ones. This experiment took insights from several prior experiments where we learned:
- Users respond better to smiley ratings
- Most users who recently stayed at a hotel are visiting Booking.com to write a review, not book again
Combining those two learnings allowed us to create a larger review CTA that used smileys to encourage users to review hotels they recently stayed at.
This experiment is also another great lesson in how small changes can often yield big results.