Tag Archives: within-subject study

design: a bit behind the ball

I just had a citation alert to this article on design efficiency in DCEs in health. Nowadays I skim citation alerts (at best) as they come so thick and fast (*polishes halo*). However, this one caught my eye being in a BMJ Open Access journal and being on design, a subject currently close to my heart.

The article wasn’t bad. I just wouldn’t say it was good either. Whilst the quantity of references was sufficient, there were a number of them that frankly were irrelevant and should have been replaced by (much) more important ones. When none of the major textbooks that include chapters on design guidance (ones by Rose/Hensher/Louviere etc) are mentioned, nor a key paper by Rose and Bliemer, you sigh.

Plus, I know I might be mis-remembering this (no longer having institutional access to check, and with the paper copies of key references being packed away unaccessible at the moment), but investigations of factors affecting design efficiency have been done already, surely?

But it was the cognitive vs statistical efficiency issue that really got me to sign up in order to make the following comment (which seems, at present, to be in moderation purgatory, though Monday may change things).

Nice investigation but I’m afraid some key non-health references are missing which would have addressed/begun to address some issues you raised. Regarding design guidance, the two seminal textbooks are not referenced, together with Rose and Bliemer’s 2009 paper.

You also appear to have understated the seriousness of the problem if the quest for efficiency leads respondents to use heuristics: your results become BIASED (useless). You say “Using a statistically efficient design may result in a complex DCE, increasing the cognitive burden for respondents and reducing the validity of results. Simplifying designs can improve the consistency of participants’ choices which will help yield lower error variance, lower choice variability, lower choice uncertainty and lower variance heterogeneity” but these are the least of your worries if the functional form of the utility function depends on the design. To their credit, Rose and Bliemer pointed out this possibility back in 2009; it’s already been observed in between-subject comparisons and I and co-authors published the first within-subject study in health and found the problem was extremely severe:

Flynn TN, Bilger M, Malhotra C, Finkelstein EA. Are Efficient Designs Used In Discrete Choice Experiments Too Difficult For Some Respondents? A Case Study Eliciting Preferences for End-Of-Life care. Pharmacoeconomics 2016:34(3);273-284

The paper was submitted right around the time ours came out in the print version, but I know our e-version was around before then, not to mention the possibility of adding it at the review stage. Which actually leads me to worry about the refereeing process just as much as aspects of the original paper.

highly efficient DCEs can be bad paper published

Sorry, I should have posted this over a week ago – have had a cold and been working on a project.

But the paper on the perils of highly efficient designs in DCEs has been published! Wahay.

Personally I consider this paper to be the second best of my career (after the first BWS one in JHE)…ironic that I have left academia!

In terms of the content, some economists still don’t “get it” but that’s their problem really 😉

perils of efficient design paper accepted

Well, this it turning out to be a good week. First I hear the BWS book was published on time.

Now a paper that two of the global experts on discrete choice experiments found ground-breaking has been accepted for publication! The paper had previously been rejected by a another good journal – a decision I (and, crucially, others) found unfair since the two referee reports did not appear to be recommending rejection (though one was fairly critical and the other didn’t really “get” a lot of what we did, it being work that challenged some tenets of the current economic orthodoxy).

Anyway the paper is called “Are Efficient Designs Used In Discrete Choice Experiments Too Difficult For Some Respondents? A Case Study Eliciting Preferences for End-Of-Life Care” by T.N. Flynn, Marcel Bilger, Chetna Malhotra and Eric Finkelstein and has been accepted by Pharmacoeconomics.

The background to the paper and its implications will form the second blog in the series I am writing about the future of health state valuation using DCEs. Suffice to say, it is revolutionary because we got respondents to answer TWO DCEs, which differed radically in their statistical efficiency – one was 100% efficient, the other 40-50% efficient. The theoretical advantages of the former were, however, heavily attenuated by the fact many respondents resorted to heuristics that didn’t represent their true decision rule – causing biased estimates. Twas a cool paper, even if I say so myself. Anyway I shall follow up on this later in the week, that’s enough for today.