Search the Elsmar Cove!
**Search ALL of Elsmar.com** with DuckDuckGo including content not in the forum - Search results with No ads.
Profound Statistical Concepts

Profound Statistical Concepts 2019-10-31

Ronen E

Problem Solver
Staff member
Super Moderator
#41
Although there is - hopefully - no new 'learning' from a confirmatory study, there is still immense value in demonstrating that we got it right.
Just to clarify, I fully agree with this statement. I brought up the issue in the context of Science (or scientific research) and the argument that the purpose of science is to accumulate new knowledge or understanding. There are other types of value to be gained, definitely in industry and commerce. Technically speaking, confirmation is also important in pure scientific work, as there's a basic requirement that results have to be reproducible (and actually reproduced) to be widely accepted; the principle I quoted only looks at accumulation of new knowledge through a mathematical lens (and ignores cultural, social etc. aspects).

Indeed, "demonstrate" is frequently used in both pharma and medical devices. Regulators use it, meaning you have to demonstrate it to them. But some developers also use it, because they assume it is a foregone conclusion. From a meeting to decide how long patients should be followed in a clinical trial in order to adequately assess safety and effectiveness of a particular type of medical device:

"Some people think 12 months of data are needed. Some people think at least 6 months. Others think only 3 months will be sufficient. And some people seem to think they already know it is safe and effective, without ever having collected any data at all."
I think that there's a subtle and very important nuance here. In a clinical trial aimed at showing (for lack of a better word; I don't want to use "demonstrate" here because that's the source of the nuance) safety and effectiveness, the developers hope for a "safe and effective" outcome; they don't know it upfront because it wasn't ever shown yet. In that sense it's not a demonstration, it's an experiment (hopefully not with a 50/50 success chance!). I think that the word "demonstrate" is used in regulated products development in a different meaning: to show something to a "high degree" of certainty (not well-defined). It stems from a wide (though not-politically-acceptable, so toned-down) understanding that full certainty or proper proof is not economically viable.

Point well taken, but expectations are not knowledge.
True. An experiment generating the outcome that it was designed to (hopefully) generate is still producing new knowledge; only not a lot (per resources invested).

Is it dead in scientific journals, which is where this discussion started? I don't read a lot of scientific journals these days, but still some. I read a lot more medical journals. In the journals I read, the p-value seems to be very much alive and well. If you call being reported in a medical journal living. :p
I don't read a lot of scientific journals either; hopefully @Bev D could comment on that. Personally, I prefer books, especially by researches who have been around long enough to have a broad perspective, preferably books that have been revised (and refined) through several editions.

Okay, so maybe the p-value is not dead yet, but it's dying. These things take time. Especially in the medical field, which is conservative and slow to change by nature (a good thing most of the time, I think).
 
#42
maybe the p-value is not dead yet, but it's dying. These things take time .
Maybe, but it's been foretold for at least half a century, so color me skeptical. So much of scientific and medical research is not for knowledge, but simply for publication. I don't think the long-standing rumors about scientific journals moving away from p-values have been inspired by a better idea, but by a hope that this would make it easier to publish. At this point, it has become so easy to publish, why even bother to move away from it?
 

Ronen E

Problem Solver
Staff member
Super Moderator
#43
I don't think the long-standing rumors about scientific journals moving away from p-values have been inspired by a better idea, but by a hope that this would make it easier to publish.
To me the main point is not so much what scientific journals do or don't do. I think @Steve Prevette mentioned it only to show that this is already a widespread trend rather than an anecdote. The methods I was talking of are more mathematically sound / more rational, not necessarily easier to implement or easier to get an idea across the line through, so I don't see how their use would make it "easier to publish". I thought they are taking hold simply because they are theoretically superior.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#44
The statistical community is pushing for it’s elimination because it is bogus and it is too often misused in order to get published or ‘prove’ dubious claims. The intent is to make it MORE difficult to get published. Doing the actual work to actually determine something is work, typing in “(p value <.05)” is not.
 
#45
I think science would be well served...and open to...better statistical tools. The world of scientific (and medical) publishing is another matter.

Even the textbook has some sidebar discussions of certain scientific journals moving away from P-values.
I take "even" and "certain" as indicators that it is not widespread. I take "moving away from" as an indicator that p-values is where they are at, since that is the point where you move from. But mostly I take it that this is literally the same way the situation has been described for decades, so not an indicator of much movement, but instead continued use of hopeful language. If the textbook didn't go so far as to identify any of the "certain" journals, maybe more like a prayer.
 

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#46
OK, if I have to I'm going to pull out the textbook and quote the sidebar exactly. Don't know why that is necessary but will do once get home.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#48
Watchcat; I'm still not sure of your point. You seem to agree that the p value & null hypothesis approach is not the best. You seem to be narrowing your complaints(?) to your perception that the scientific and medical journals simply aren't changing and have no intent to do so.

The people who are involved in this thread are trying their best - and being fairly successful - in changing their worlds. They come here to help spread that change to other worlds and although we haven't changed the entire world together we are changing a large part of it. My position has always been that if you aren't part of the fight for change you are part of the resistance to change. You seem to be giving the people who you seem to agree with a really hard time - Why?
 
#49
I don't think it is necessary, but I'd certainly be interested to know which journals have moved away from p-values, and, based on this discussion, now equally interested to know what they have moved towards.

Watchcat; I'm still not sure of your point.
Same as when Bev D inquired: Understanding. Mine, not yours.

I have expressed no complaints that I know of, but I've found that whether or not something is a complaint is often in the eye of the beholder.

if you aren't part of the fight for change you are part of the resistance to change
I'm deeply opposed to fighting for change. I subscribe to "Be the change" instead. If I were to have the opportunity to use something better than a p-value, I'd be very happy to do so...would in fact come straight here for suggestions, but that strikes me as unlikely. Regardless, the world is overrun with things that would benefit from changing. These journals belong to the scientific and medical communities. If they want them to change, they will. If they don't, they won't. In any case, sounds like the statistical community has got this. I'm quite sure I could not add anything at all useful to their efforts.

I do have a project I call "Bad Evidence," which focuses on any publications related to the medical device industry and device regulation. One has a working title of "Four Fools from Harvard and a Whole Bunch of Idiots at the BMJ." (I'll be as interested as anyone to see whether I have the nerve to go live with this title, LOL.) Would love some help with it from the statistical gurus here at Elsmar, as linear regression is not my strong suit, but there is only so much even a guru can do when an analysis is poorly and incompletely reported.

You seem to be giving the people who you seem to agree with a really hard time
I can't speak for them, but my personal impression is that everyone involved in this discussion has been able to articulate their perspectives without breaking a sweat. If I didn't have the impression that there was that level of expertise to be found here, I'd be in another room.
 
Last edited:
Top Bottom