Profound Statistical Concepts

Profound Statistical Concepts 2019-10-31

Ronen E

Problem Solver
Staff member
Moderator
#41
Although there is - hopefully - no new 'learning' from a confirmatory study, there is still immense value in demonstrating that we got it right.
Just to clarify, I fully agree with this statement. I brought up the issue in the context of Science (or scientific research) and the argument that the purpose of science is to accumulate new knowledge or understanding. There are other types of value to be gained, definitely in industry and commerce. Technically speaking, confirmation is also important in pure scientific work, as there's a basic requirement that results have to be reproducible (and actually reproduced) to be widely accepted; the principle I quoted only looks at accumulation of new knowledge through a mathematical lens (and ignores cultural, social etc. aspects).

Indeed, "demonstrate" is frequently used in both pharma and medical devices. Regulators use it, meaning you have to demonstrate it to them. But some developers also use it, because they assume it is a foregone conclusion. From a meeting to decide how long patients should be followed in a clinical trial in order to adequately assess safety and effectiveness of a particular type of medical device:

"Some people think 12 months of data are needed. Some people think at least 6 months. Others think only 3 months will be sufficient. And some people seem to think they already know it is safe and effective, without ever having collected any data at all."
I think that there's a subtle and very important nuance here. In a clinical trial aimed at showing (for lack of a better word; I don't want to use "demonstrate" here because that's the source of the nuance) safety and effectiveness, the developers hope for a "safe and effective" outcome; they don't know it upfront because it wasn't ever shown yet. In that sense it's not a demonstration, it's an experiment (hopefully not with a 50/50 success chance!). I think that the word "demonstrate" is used in regulated products development in a different meaning: to show something to a "high degree" of certainty (not well-defined). It stems from a wide (though not-politically-acceptable, so toned-down) understanding that full certainty or proper proof is not economically viable.

Point well taken, but expectations are not knowledge.
True. An experiment generating the outcome that it was designed to (hopefully) generate is still producing new knowledge; only not a lot (per resources invested).

Is it dead in scientific journals, which is where this discussion started? I don't read a lot of scientific journals these days, but still some. I read a lot more medical journals. In the journals I read, the p-value seems to be very much alive and well. If you call being reported in a medical journal living. :p
I don't read a lot of scientific journals either; hopefully @Bev D could comment on that. Personally, I prefer books, especially by researches who have been around long enough to have a broad perspective, preferably books that have been revised (and refined) through several editions.

Okay, so maybe the p-value is not dead yet, but it's dying. These things take time. Especially in the medical field, which is conservative and slow to change by nature (a good thing most of the time, I think).
 
Elsmar Forum Sponsor

Watchcat

Trusted Information Resource
#42
maybe the p-value is not dead yet, but it's dying. These things take time .
Maybe, but it's been foretold for at least half a century, so color me skeptical. So much of scientific and medical research is not for knowledge, but simply for publication. I don't think the long-standing rumors about scientific journals moving away from p-values have been inspired by a better idea, but by a hope that this would make it easier to publish. At this point, it has become so easy to publish, why even bother to move away from it?
 

Ronen E

Problem Solver
Staff member
Moderator
#43
I don't think the long-standing rumors about scientific journals moving away from p-values have been inspired by a better idea, but by a hope that this would make it easier to publish.
To me the main point is not so much what scientific journals do or don't do. I think @Steve Prevette mentioned it only to show that this is already a widespread trend rather than an anecdote. The methods I was talking of are more mathematically sound / more rational, not necessarily easier to implement or easier to get an idea across the line through, so I don't see how their use would make it "easier to publish". I thought they are taking hold simply because they are theoretically superior.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#44
The statistical community is pushing for it’s elimination because it is bogus and it is too often misused in order to get published or ‘prove’ dubious claims. The intent is to make it MORE difficult to get published. Doing the actual work to actually determine something is work, typing in “(p value <.05)” is not.
 

Watchcat

Trusted Information Resource
#45
I think science would be well served...and open to...better statistical tools. The world of scientific (and medical) publishing is another matter.

Even the textbook has some sidebar discussions of certain scientific journals moving away from P-values.
I take "even" and "certain" as indicators that it is not widespread. I take "moving away from" as an indicator that p-values is where they are at, since that is the point where you move from. But mostly I take it that this is literally the same way the situation has been described for decades, so not an indicator of much movement, but instead continued use of hopeful language. If the textbook didn't go so far as to identify any of the "certain" journals, maybe more like a prayer.
 

Steve Prevette

Deming Disciple
Staff member
Super Moderator
#46
OK, if I have to I'm going to pull out the textbook and quote the sidebar exactly. Don't know why that is necessary but will do once get home.
 

Marc

Hunkered Down for the Duration with a Mask on...
Staff member
Admin
#47
OK, if I have to I'm going to pull out the textbook and quote the sidebar exactly. Don't know why that is necessary but will do once get home.
You have my deepest respect and sincere appreciation to take the time to do so.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#48
Watchcat; I'm still not sure of your point. You seem to agree that the p value & null hypothesis approach is not the best. You seem to be narrowing your complaints(?) to your perception that the scientific and medical journals simply aren't changing and have no intent to do so.

The people who are involved in this thread are trying their best - and being fairly successful - in changing their worlds. They come here to help spread that change to other worlds and although we haven't changed the entire world together we are changing a large part of it. My position has always been that if you aren't part of the fight for change you are part of the resistance to change. You seem to be giving the people who you seem to agree with a really hard time - Why?
 

Watchcat

Trusted Information Resource
#49
I don't think it is necessary, but I'd certainly be interested to know which journals have moved away from p-values, and, based on this discussion, now equally interested to know what they have moved towards.

Watchcat; I'm still not sure of your point.
Same as when Bev D inquired: Understanding. Mine, not yours.

I have expressed no complaints that I know of, but I've found that whether or not something is a complaint is often in the eye of the beholder.

if you aren't part of the fight for change you are part of the resistance to change
I'm deeply opposed to fighting for change. I subscribe to "Be the change" instead. If I were to have the opportunity to use something better than a p-value, I'd be very happy to do so...would in fact come straight here for suggestions, but that strikes me as unlikely. Regardless, the world is overrun with things that would benefit from changing. These journals belong to the scientific and medical communities. If they want them to change, they will. If they don't, they won't. In any case, sounds like the statistical community has got this. I'm quite sure I could not add anything at all useful to their efforts.

I do have a project I call "Bad Evidence," which focuses on any publications related to the medical device industry and device regulation. One has a working title of "Four Fools from Harvard and a Whole Bunch of Idiots at the BMJ." (I'll be as interested as anyone to see whether I have the nerve to go live with this title, LOL.) Would love some help with it from the statistical gurus here at Elsmar, as linear regression is not my strong suit, but there is only so much even a guru can do when an analysis is poorly and incompletely reported.

You seem to be giving the people who you seem to agree with a really hard time
I can't speak for them, but my personal impression is that everyone involved in this discussion has been able to articulate their perspectives without breaking a sweat. If I didn't have the impression that there was that level of expertise to be found here, I'd be in another room.
 
Last edited:
Thread starter Similar threads Forum Replies Date
W Deming's SoPK (System of Profound Knowledge) Discussion Philosophy, Gurus, Innovation and Evolution 220
W Deming's SoPK (System of Profound Knowledge) Challenge Philosophy, Gurus, Innovation and Evolution 66
S How to perform verification of the Statistical Analysis Software? Qualification and Validation (including 21 CFR Part 11) 2
M V&V phase: Justification of acceptance criteria (statistical method ) - (Medical Device) Design and Development of Products and Processes 2
Bev D Statistical Alchemy Misc. Quality Assurance and Business Systems Related Topics 1
K Looking for guidance to write an SOP on Statistical Methodologies? Statistical Analysis Tools, Techniques and SPC 7
M Minimum sample size - Guidance and statistical rationale Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
N Design Verification & Process Validation - Statistical sample sizes Design and Development of Products and Processes 2
John Predmore Interactive visualization through graphical simulation of statistical concepts Statistical Analysis Tools, Techniques and SPC 3
A Statistical Analysis - Check if these organisms at different concentrations affect the growth of wheat seedlings Using Minitab Software 4
H Statistical Techniques Procedure - What should be included Document Control Systems, Procedures, Forms and Templates 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
optomist1 It’s time to talk about ditching statistical significance Statistical Analysis Tools, Techniques and SPC 6
Marc Steve Prevette's Statistical Process Control (SPC) "Library" Statistical Analysis Tools, Techniques and SPC 0
John Predmore A Balanced view of statistical tests Statistical Analysis Tools, Techniques and SPC 3
V Statistical basis and justification while comparing / changing sampling plans Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
S SPC (Statistical Process Control) for Unilateral Tolerance - Questions Statistical Analysis Tools, Techniques and SPC 6
S IATF 16949 9.1.1.3 Application of statistical concepts - Our technicians are quizzed for statistical knowledge IATF 16949 - Automotive Quality Systems Standard 3
K Please help identify appropriate statistical treatment Statistical Analysis Tools, Techniques and SPC 13
ScottK Statistical basis for 30 pieces for FAI 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7
B IATF 16949 clause 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
A Statistical Process Control and Inspection in Footwear Production Statistical Analysis Tools, Techniques and SPC 0
M IATF 16949 Cl. 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
Steve Prevette Statistical Process Control Library Statistical Analysis Tools, Techniques and SPC 17
Marc Happy Birthday Statistical Steven - 2015 Covegratulations 10
L When are Statistical techniques not applicable? Service Industry Specific Topics 16
M FDA 21 CFR 820.250 - Does "valid statistical" always mean math? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 6
R Common Statistical Errors Using Minitab Software 1
Y Statistical Analysis of Road Traffic Data Statistical Analysis Tools, Techniques and SPC 11
B Class II Medical Device Manufacturer - SOP for 820.250 Statistical 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
E Correct Statistical Test comparing 2 Groups Statistical Analysis Tools, Techniques and SPC 14
A Statistical Software Calibration using Ford's "Sample Calibration File" Statistical Analysis Tools, Techniques and SPC 8
J Defining Martial Arts and Gymnastics Statistical Techniques Statistical Analysis Tools, Techniques and SPC 4
J Capability Analysis - Unusual Statistical Distribution of my Proccess Capability, Accuracy and Stability - Processes, Machines, etc. 5
M PhD Thesis Data Statistical Analysis Methods Statistical Analysis Tools, Techniques and SPC 2
J Statistical Significance and SPC Control Chart Reports Statistical Analysis Tools, Techniques and SPC 9
N Statistical Quality Improvement Action for Small Batch Production Statistical Analysis Tools, Techniques and SPC 17
V Validation of macro - scripts - programs used in statistical software (Minitab-SAS... Qualification and Validation (including 21 CFR Part 11) 5
Moncia Statistical Process Control Crash Course - Question Quality Manager and Management Related Issues 10
H Statistical Models for Predictive Management of Software Processes Software Quality Assurance 2
A Statistical Correlation between ordered SKUs Statistical Analysis Tools, Techniques and SPC 8
S Minitab and Crystal Ball Statistical Analysis Software Using Minitab Software 13
M Determining if two different X's have any Statistical Significance on the Y's Statistical Analysis Tools, Techniques and SPC 4
I Statistical Stability for the PQ of Analytical Equipment Qualification and Validation (including 21 CFR Part 11) 1
F Statistical Comparison of Product: High Average vs. Low Range Capability, Accuracy and Stability - Processes, Machines, etc. 13
E Using ANOVA during the PQ Validation Run to evaluate Statistical Differences Statistical Analysis Tools, Techniques and SPC 4
W Gage R&R for gage pins used to inspect a hole ID called a Statistical Tolerance Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 3
B AS 9100C sec 8.2.4 - " Recognized Statistical Principles" meaning AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 7
R What is PSW - Statistical Process Package + Level 5 APQP and PPAP 7
O Is SPC (Statistical Process Control) always required? Statistical Analysis Tools, Techniques and SPC 4

Similar threads

Top Bottom