Venue: The Fuqua School of Business, Duke University, 1 Towerview Drive, Durham, NC 27708-0120
Presentation
Are Stated-Preference Experimental Designs That Are Worth Doing Necessarily Worth Doing Well?
BACKGROUND: Constructing statistically efficient experimental designs is fundamental to the practice of conjoint analysis or discrete-choice experiments. Experimental design refers to the choice of attributes and attributes levels and how they are combined to create alternatives in choice sets. The efficiency of resulting estimates depends largely on the characteristics of the design. Recent literature identifies strategies for constructing experimental designs that range from simple random pairing of catalog plans to more complicated and time-consuming D-optimal and cyclical designs. While statistical efficiency is an important goal in constructing designs, it is worth determining whether the efficiency gains are worth the effort of constructing an optimal design.
OBJECTIVE: This study evaluates and compares the empirical efficiency of three different experimental-design approaches: D-optimal, random-pair catalog, and cyclical designs.
METHODS: Unlike previous studies, we employ simulations based on actual, empirical preference-parameter estimates to investigate design efficiency and accuracy. The study uses random-parameters logit estimates from a study that investigated health-care payer reimbursement preferences for six drug attributes with unbalanced levels. Parameter-distribution estimates from this study are used as true priors for conditioning D-optimal designs, for simulating response data for alternative designs, and for calculating mean squared errors (MSE) for model simulations. The study first compares D-optimal and random-pair catalog designs by regressing efficiency as measured by D-scores and design performance measured by mean squared errors on design characteristics, including number of dominated pairs, number of overlaps, correlations among attribute-level differences, and choice-probability balance. We then compare the MSE efficiency of D-optimal designs to the cyclical design approach recently proposed by Street and Burgess.
RESULTS: As expected, the results indicate an inverse relationship between D-score and MSE. This relationship is strong and consistent for D-optimum designs, but weak and inconsistent for random-pair catalog designs. We find that correlations in attribute-level differences have the largest influence on the D-scores and the MSEs of both the random-pair catalog and D-optimum designs. Again as expected, this relationship is negative for D-score and positive for MSE. However, an increase in the attribute correlations has a greater negative effect on random-pair catalog designs than on D-optimum designs. On average, D-optimal designs are 88% more efficient than random-pair catalog designs, indicating that catalog-based designs would require 88% more observations to achieve the same estimation precision. Finally, depending on the sample size, we find that the D-optimal designs are between 14% and 36% more efficient than cyclical designs, given the features of the original study and using prior information on parameter values to condition the D-optimal designs.
CONCLUSIONS: Although our results are specific to a particular set of attributes, levels, and empirical preference estimates, the features of this study are more typical of the practical applications that most researchers encounter than the simplified examples often used to discuss experimental-design efficiency. The wide availability of software to construct D-optimal designs has reduced the burden on researchers of constructing efficient designs. Thus the gain in statistical efficiency appears to be well worth the relatively modest additional effort.