Tag Archives: case 2

spring cleaning

New year = digital spring cleaning time! Ugh. No matter how future-proof you aim to be with how you structure files, aim to work seamlessly across PCs etc it never takes long for reality to change and you realise you need to do the rigmarole again.

When admin is done I’m back to the project looking at comparing Case 2 BWS estimates with DCE ones. I shall look with “fresh eyes” since I haven’t worked on it since before xmas. (Plus we need to get this rounded off so we can submit and get paid, hehe.)

Then it’s the (long-delayed) big marketing push for TF Choices LTD. I’ve had a good number of proposals and funded projects come my way so far but can’t rest on my laurels…time to make sure a load of marketers and others know what I can do for them, in addition to the academic community I was part of!

I can’t think of anything methodological I want to shout about today (phew, they think)….I’ll continue to post anything big or of key relevance but as there are only so many hours in the day and company stuff must come front and centre in 2017 it’s likely that my comments and posts will be related to things I’m doing at the time (like Case 2 vs DCEs) rather than detailed posts triggered by twitter or citation alerts I get.

 

BWS neither friend nor foe

This post replies to some requests I have had asking me to respond to a paper concluding that DCEs are better than BWS for health state valuation. To be honest I am loathe to respond, for reasons that will become apparent.

First of all, let me clarify one thing that people might not appreciate – I most definitely do not want to “evangelise” for BWS and it is not the solution in quite a few circumstances. (See the papers coming out from the CHU-9D child health valuation study I was involved with for starters – BWS was effectively a waste of resources in the end….”best” choices were all we could use for the tariff.)

I only really pushed BWS strongly in my early days as a postdoc when I wanted to make a name for myself. If you read my papers since 2007 (*all* of them) you’ll see the numerous caveats appear with increasing frequency. And that’s before we even get to the BWS book, where we devote an entire chapter discussing unresolved issues including the REAL weaknesses and research areas for BWS (as opposed to straw men I have been seeing in recent literature).

OK now that’s out of the way, I will lay some other cards on the table, many of which are well-known since I’ve not exactly been quiet about them. I had mental health issues associated with my exit from academia. I’m back on my feet now doing private sector work for very appreciative clients, but that doesn’t mean I want to go back and fight old battles….battles which I erroneously thought us three book authors had “won” by passing muster with the top mathematical psychologists, economists and others in the world during peer review. When you publish a paper in the Journal of Mathematical Psychology (the JHE of that field) illustrating a key feature/potential weakness of a DCE (or specifically Case 2 BWS) back in 2008 you tend to expect that papers published in 2016 would not ignore this and would not do research that showed zero awareness of this issue and as a result made fundamental errors – after all, whilst we know clinical trials take a while to go from proposal to main publication, preference studies do NOT take 8+ years to go through this process. I co-ran a BWS study from conceptualisation to results presentation in 6 days when in Sydney. Go figure.

So that’s an example of my biggest frustration – the standards of literature review have often been appalling. Two or three of my papers (ironically including the JHE one, which includes a whopping error which I myself have repeatedly flagged up and which I corrected in my 2008 BMC paper) seem to get inserted as “the obligatory BWS reference to satisfy referees/editors” and in many cases bear no relation to the point being made by authors. Alarm bells immediately flash when I read an abstract via a citation alert and see those were my references. But it keeps happening. Not good practice, folks.

In fact (and at a recent meeting someone with no connection to me said the same thing) in certain areas of patient outcomes research the industry reviews are considered far better than academic ones – they have to be or get laughed out of court.

Anyway, I have been told that good practice eventually drives out bad. Sorry, if that’s true, the timescale was simply too long for me, which didn’t help my career in academia and raised my blood pressure.

Returning to the issue at hand. I’m not going to go through the paper in question, nor the several others that have appeared in the last couple of years purporting to show limitations of BWS. I have a company to run, caring obligations and I’ve written more than enough for anyone to join the dots here if they do a proper literature review. My final attempt to help out was an SSRN paper. But that’s it – without some give and take from the wider community, my most imaginative BWS work will be for clients who put food on the table and who pay – sometimes quite handsomely – for a method that when properly applied shows amazing predictive ability together with insights into how humans make decisions.

Now, of course, health state valuation is another kettle of fish – no revealed preference data etc. However, Tony, Jordan and I discussed why “context” is key in 2008 (JMP); I expounded on this with reference to QALYs in my two 2010 single authored papers, and published a (underpowered) comparison in the 2013 JoCM paper (which I first presented at the 2011 ICMC conference in Leeds, getting constructive criticism from the top choice modellers on Earth). So this issue is not particularly new.

It’s rather poor that nobody has actually used the right design to compare Case 2 BWS with DCEs for health state valuation…I ended up deciding “if you want something done properly you have to do it yourself” and I am very grateful to the EuroQoL Foundation for funding such a study, which I am currently analysing with collaborators. I don’t really “have a dog in this fight” and if Case 2 proves useful then great, and if not then at least I will know exactly why not…and the reasons will have nothing to do with the “BWS is bad m’kayyyyy” papers published recently. (To be fair, I am sometimes limited in what I can access, with no longer having an academic affiliation so full texts are sometimes unavailable, but when there’s NO mention of attribute importance in the abstract, NOR why efficient designs for Case 2 are problematic my Bayesian estimate is 99.99% probability the paper is fundamentally flawed and couldn’t possibly rule BWS in or out as a viable competitor to a DCE.)

If you’d like to know more:

  • Read the book
  • Read all the articles – my google scholar profile is up to date
  • Get up to speed on the issues in discrete choice design theory – fast. Efficient designs are in many many instances extremely good (and I’ve used them) but you need to know exactly why in a Case 2 context they are inappropriate.

If you still don’t understand, get your institution to contract me to run an exec education course. When I’m not working, I’m not earning, full stop.

I’m now far more pragmatic about the pros and cons of academia and really didn’t want to be the archetypal “I’m leaving social media now” whinger. And I’m not leaving. But I am re-prioritising things. Sorry if this sounds harsh/unhelpful – I didn’t want to write this post and hoped to quietly slip beneath the radar, popping up when I thought something insightful based on one of BWS’s REAL disadvantages or Sen’s work etc was mentioned. But people I respect have asked for guidance. So I am giving what I can, given 10 minutes free time I have.

Just trying to end on a positive note – I gave a great exec education course recently. It was a pleasure to engage with people who asked questions that were pertinent to the limitations of BWS and who just wanted to use the right tool for the right job. That’s what I try to do and what we should all aim for. I take my hat off to them all.

case 2 versus case 3

I have a few commitments to finish in terms of academia. (Actually, and technically, I don’t, but there are some I enjoy and want to finish off.)

Today through Friday I am going through some interesting data of a colleague that compares Case 2 (Profile Case) Best-Worst Scaling (BWS) with multiprofile traditional discrete choice experiment data (BWS Case 3 and DCEs).

It is has been known since 2008 how and why estimates from these different methods might differ (see Marley, Flynn & Louviere, JMP 2008 and Flynn et al, JoCM 2013). Yet unfortunately there remains a lot of misunderstanding out there on this issue. Fortunately this colleague asked for help at an early stage and we have been interrogating some interesting data for a year now. (I hate it when people choose to ignore the people who could actually help them explain these things….but 95% of the time it rebounds on them as their papers become cited for the wrong reasons….ego ego ego…..)

So people who have been collegiate and come talk have got a lot from me (I hope!) Plus I really hope it helps their academic careers, particularly when results such as those I am seeing this week arise 🙂

Anyway, if you want to know more about this issue – and it has HUGE implications for ANYBODY working in health or applied economics – you should buy the book (yes another plug….)

In other news, a paper I co-authored which would have ruffled a LOT of feathers got rejected on monday. Strangely enough, the referee reports did not give the impression (to me and my most senior co-author) that they thought it should be rejected – granted, some heavy revisions were suggested (which I thought largely were perfectly fair) – but the particular editor who dealt with it gave a flat out reject.

Strange that a study heavily (and independently) endorsed by two of the top experts in DCEs on the planet got a flat out rejection….but I suspect academic politics were in play here….but, unlike in the past, my blood pressure did not rise. This isn’t my problem anymore. The paper would ultimately have got a sh*tload of citations in my opinion (beating my most cited paper) but that’s the journal’s loss. *I* know what the problem is that my study identified so I will utilise the implications in my commercial studies 🙂