Tag Archives: textbooks

design: a bit behind the ball

I just had a citation alert to this article on design efficiency in DCEs in health. Nowadays I skim citation alerts (at best) as they come so thick and fast (*polishes halo*). However, this one caught my eye being in a BMJ Open Access journal and being on design, a subject currently close to my heart.

The article wasn’t bad. I just wouldn’t say it was good either. Whilst the quantity of references was sufficient, there were a number of them that frankly were irrelevant and should have been replaced by (much) more important ones. When none of the major textbooks that include chapters on design guidance (ones by Rose/Hensher/Louviere etc) are mentioned, nor a key paper by Rose and Bliemer, you sigh.

Plus, I know I might be mis-remembering this (no longer having institutional access to check, and with the paper copies of key references being packed away unaccessible at the moment), but investigations of factors affecting design efficiency have been done already, surely?

But it was the cognitive vs statistical efficiency issue that really got me to sign up in order to make the following comment (which seems, at present, to be in moderation purgatory, though Monday may change things).

Nice investigation but I’m afraid some key non-health references are missing which would have addressed/begun to address some issues you raised. Regarding design guidance, the two seminal textbooks are not referenced, together with Rose and Bliemer’s 2009 paper.

You also appear to have understated the seriousness of the problem if the quest for efficiency leads respondents to use heuristics: your results become BIASED (useless). You say “Using a statistically efficient design may result in a complex DCE, increasing the cognitive burden for respondents and reducing the validity of results. Simplifying designs can improve the consistency of participants’ choices which will help yield lower error variance, lower choice variability, lower choice uncertainty and lower variance heterogeneity” but these are the least of your worries if the functional form of the utility function depends on the design. To their credit, Rose and Bliemer pointed out this possibility back in 2009; it’s already been observed in between-subject comparisons and I and co-authors published the first within-subject study in health and found the problem was extremely severe:

Flynn TN, Bilger M, Malhotra C, Finkelstein EA. Are Efficient Designs Used In Discrete Choice Experiments Too Difficult For Some Respondents? A Case Study Eliciting Preferences for End-Of-Life care. Pharmacoeconomics 2016:34(3);273-284

The paper was submitted right around the time ours came out in the print version, but I know our e-version was around before then, not to mention the possibility of adding it at the review stage. Which actually leads me to worry about the refereeing process just as much as aspects of the original paper.