A bias correction approach for interference in ranking experiments

Goli, A, Lambrecht, A and Yoganarasimhan, H (2024) A bias correction approach for interference in ranking experiments. Marketing Science, 43 (3). pp. 590-614. ISSN 0732-2399 OPEN ACCESS

Abstract

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the Total Average Treatment Effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased due to interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests, even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, i.e., items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.

More Details

Item Type: Article
Subject Areas: Marketing
Additional Information:

This is the accepted version of a manuscript which has been accepted for publication at Marketing Science. It can be accessed via the DOI: https://doi.org/10.1287/mksc.2022.0046

Date Deposited: 16 Oct 2023 12:46
Date of first compliant deposit: 28 Jun 2023
Subjects: Experiments
Last Modified: 05 Nov 2024 02:31
URI: https://lbsresearch.london.edu/id/eprint/2936
More

Export and Share


Download

Accepted Version - Text

Statistics

Altmetrics
View details on Dimensions' website

Downloads from LBS Research Online

View details

Actions (login required)

Edit Item Edit Item