Tag Archives: references

BWS correct referencing redux

This is not exactly a moan (since in some cases I’m requesting fewer references to one or two of my own papers, which is all very nice!). It’s just a reminder that BWS has been an evolving technique over many years and I continue to note too many people just seem to add the JHE 2007 paper as “the BWS reference” when it really isn’t supporting what they are doing or saying.

I’m not been afraid to admit when I’ve done something incorrect/misleading, or when the field has moved on and an earlier paper is becoming outdated. (So when I call others on bad referencing, rest assured that I do the same for myself.)

Some points to note:

  • The JHE article was the first comprehensive explanatory Profile Case (Case 2) BWS paper. However, the “marginal models” there involved coding that although gives correct point estimates, give misleading summary statistics like log-likelioods, by not taking account of the sequential nature of the data. Thus, a choice from 5, means only 4 options are available for the second choice.
  • This was corrected ASAP – the 2008 BMC paper on dermatology study corrected this, so marginal sequential models should really reference this paper.
  • References to “dual/multi stage choice tasks” (primarily to get QALYs) should start with my 2010 Pharmacoeconomics paper, since that was the first to propose these (including the DCE+TTO rescaling) method. Too many researchers reference later papers.
  • I was also first in explaining why the “death state” can’t be valued in a DCE without duration and a higher resolution design – in 2008 I wrote about this in Pop Health Metrics, with the God of math psych, Tony Marley, amongst others. I also pointed out why variance scale factors can be highly problematic in DCEs/other choice models. I certainly wasn’t first on the latter point – you should be looking to papers in the 1990s by Swait & Louviere, and Hensher and Louviere for that.
  • First reference to a Case 1 BWS study is in The Patient: Patient-Centered Outcomes Research (2010) by Louviere and Flynn (to my knowledge – I am happy to be corrected if wrong).
  • If you’re comparing Case 2 BWS with DCEs you really should be understanding and discussing how they differ, which was introduced in detail in the 2013 JoCM paper by Flynn et al. Subsequent discussion in the book (2015). DO NOT conclude that either method is “wrong”/”right” purely on basis of comparison of results from each task. Our work explains why they might differ.
  • For Case 3 BWS I’m not the key person, Emily Lancsar was/is big in introducing and applying this in health. Please also note the correct name for this is the “multi-profile case” as agreed by Louviere, Marley and me in preparation for the book. Like the profile case, renaming was done so as to better describe what made Cases 2 and 3 distinct from other Cases.
  • First reference to a peer-reviewed published Case 2 study was from the 1990s by Szeinbach et al; first UK study was 2006 by our team in BJD.
  • Finally, the emerging problems with highly efficient designs: Rose and Bliemer hypothesised this back in 2009; I and team published the first within-subject confirmation in Pharmacoecon 2016.

Thus, it’s just a guide to help practitioners get the correct reference for BWS and associated conceptual issues. Hope it helps. I may add to this if I think of other issues that are incorrectly attributed.


what I will review

Note to journal editors etc.

Since I have now moved to industry, reviewing articles is no longer part of my duties.

Because publishing isn’t entirely off my radar, I will do a little bit of refereeing. I have one editorial position for which I shall continue some reviewing duties, but otherwise I shall only be accepting articles that:

(1) Are genuinely innovative, not simply “me-too” choice modelling articles;

(2) Have had their referencing adequately checked by editors. There are key references for choice modelling: the “giants” being Thurstone, McFadden, Luce, Hensher, Louviere, Swait, Marley, Rose (and yours truly for a lot of best-worst scaling development), that are primary sources. An article with none of these and a bibliography with 5-9 references from the health literature would make me highly sceptical that an adequate literature review has been conducted.

For editors in health services research and allied medicine, I understand that the number of people with both choice modelling and health care skills is small. I would therefore suggest you split reviewing duties between a “methodological” referee and a “health” referee in order to better review articles. Methodological reviewers can be found across multiple fields and the major choice modelling textbooks have a multitude of these in the bibliography.

Kind regards,


a plea for better citations

OK a moan for this first day back after the long weekend. (I feel entitled to since I have been productive on the blogging front recently!)

It concerns citations – or the lack of them. I was recently made aware that there are a bunch of recent papers out there which have tried to claim best-worst scaling as their own – incorrect or no references are made to the work of Jordan Louviere (who invented it) or of me (the guy who did most to develop it in health care). Instead, some tweaks have been made – which have not been proved to have any theoretical basis, nor to even be mathematically meaningful by mathematical psychologists – in order to try to sell the methods as “novel and these researchers’ own”. Some of these tweaks simply make no sense to me and I don’t really understand what the hell is trying to be done. Others, I know to be a complete waste of time and indeed, if I were to be bad-minded, appear to be attempts to use trendy statistical methods to confuse and impress more gullible or less experienced editors.

I will simply give one example of something ludicrous.

Case 1 Best-Worst Scaling (BWS) attempts to value a list of items on a probabilistic scale. If you administer a balanced incomplete block design to every respondent, you can then plot the empirical distribution of preferences for every item across the entire sample. So, just so we are clear, there is absolutely NO rationale for assuming any between-individual distribution. You simply “look” at the distribution by plotting it. Using an empirical bayesian or frequentist model which assumes any kind of distribution is just plain idiotic and a waste of paper (or pixels). Plus I would question exactly why you were doing this, given that I have given a zillion presentations showing how to look at empirical distributions, there are published papers demonstrating this etc etc etc.

So, referees and editors:
(1) Can you apply the same standards of citations and referencing that are applied to RCTs and other studies in health?
(2) If the authors use a method that has not been given the seal of approval by a mathematical psychologist or someone with equivalent expertise, can you ask them to provide the theorems and proofs as to why their methods actually give what they claim to give.
(3) Discrete choice models are rooted in theory – if authors are going to throw that out the window by marrying these with atheoretical techniques they better have a VERY good explanation in their discussion as to why they had to do this. Don’t get me wrong, I’m pragmatic, and sometimes you do need to resort to this. But you better have a damn good explanation for why you had to do so.

Finally, some fun. We all know how this turns out don’t we?

Spiders+nuclear reaction = ???

Spiders+nuclear reaction = ???