Power and Validity in Pharmco Claims
One thing I did in my graduate school days was to take a lot of courses in statistics. Yes, I know that you are wondering: "Why?"Simply put, my curiosity was piqued and I was determined to understand the basis of the "lies, damn lies and statistics" that are commonly quoted and thrown at the public as evidence for one claim or another. I took at least four or five courses in stat, including three courses in advanced stat in the internationally recognized Graduate Psychology Department at the University of Memphis. Of course, I cannot recall all the formulas and mathematical gyrations without looking back at a book, but I CAN recall some fairly straightforward principles I took with me. These principles serve as kind of an informal "BS detector" when I read or see claims made by politicians, advertisers, drug reps, and, yes, doctors and scientists [the majority of whom know as little as the rest of us about what constitutes a valid claim!].
Here they are:
(1) "Size Matters": It might seem obvious, but it is important that a LARGE enough sample size be in the test pool for a claim to be representative of what is true in the world at large. Obviously, if your pool contains 500 subjects, its outcome is more "powerful" than one with 50 subjects, all else being equal.
(2) Chance: Any formal study or research project sets what is called a probability threshold. You may see this characterized as what is called a "p" value. So, a hypothetical study may say that a given result is valid with "p<.05". What this means is that, for this study, there is a 5% chance that the allegedly valid result has been obtained by chance, that is, the finding is actually incorrect! this is called a False Positive result. Here is the take away message: ANY study has a p value of some amount or other. So, it is most important that we know what it is and recognize that the claim being made could be a random false for positive some percentage of the time less than the p value.
(3) Variance: Here is where it is important not to be misled by AVERAGE results. Any credible study should be able to report a "variance" value for any measure where it is also giving you the mean or average result. You can think of the variance value as a measure of the "spread" or distribution of the results. Lower variance means that the typical outcome for each subject in the test is closer to the average value than would be the case with a higher variance. I'll put it in every day terms for you: Let's suppose that you can take two medications and that both have the same average outcome: I'll say 80% of the time that each is effective for ten users. Which would you feel better about putting in your mouth--one that has this pattern of "percentage of the time effective" outcomes for its ten users:
70, 90, 60, 100, 80, 75, 85, 68, 92, 80, 80
or this one:
20, 80, 100, 60, 100, 60, 75, 90, 85, 95, 35.
Both have the same AVERAGE outcome of 80% effectiveness. But only a crap shooter would feel more comfortable with option number two if option number one is also available.
(4) Test dilution: This concept is a little bit more complex, but I will try to make an analogy. In any test study, there is a dilution of the validity of an outcome claimed, when the same subjects are used to conduct more than one test related to the same finding. The statistical logic is straightforward but esoteric. The short version is that there is a tradeoff between statistical validity and the number of tests we run on the same subjects. So, when an argument is made on behalf of some form of treatment, be sure to ask the claims maker about repeated testing of the same subjects. If you want to put them really on their heels, ask them if they made a "Bonferroni adjustment" in determining the outcomes.
(5) Conditional probabilities: Here we are talking about the validity of a claim made where the probability of one event is actually dependent on some other event(s). A classic example is the diagnosis and treatment of prostatic cancer in men. One variable is the rate at which this disease in one manifestation or another occurs in men as they age. The second variable is the accuracy with which it can be diagnosed by method "X". Let's suppose that your Uncle George goes to the doctor and gets a test result that says he has prostatic cancer. What is the likelihood that he actually has the disease? What should be done about it if we have a range of potential treatments that work some percentage of the time? Does it matter if most cases of prostatic cancer never become life-threatening in any case? The probabilities here are all interrelated and any claim that does not take the actual "base rates" of disease incidence and mortality or morbidity into account is NOT ACCURATE.
This is just a start. I could go on at great length, but then I'd be "teching stat" and that is not my objective. My goal is simply to play a constructive role in the never ending campaign to separate reasonable claims from what the lawyers call "puffery"-- more commonly referred to as malarkey!!
Bob Sweeney
posted by Robert E. Sweeney, DA, MS @ 11:09 AM
0 Comments:
Post a Comment
<< Home