It’s time to talk about ditching statistical significance

Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#2
I hadn’t read it yet, but certainly it isn’t the first. (Later today I’ll post some other article titles and link to the free ones). You can actually start with Deming’s 1975 article “On probability as a basis for action”.

I have been advocating for this for some. In my 35 years of solving hundreds of complex problems, running insightful MSAs, process capability studies etc. I’ve never used a p value; I never checked for Normality, etc.


The thing is that the whole null hypothesis / p value / normal centric view is a hodgepodge ritual that no founding statistician (Fisher, Pearson) ever advocated for. It doesn’t add insight, it obfuscates. It enables a lack of thought, understanding or insight. It is black box. Just read most of the postss here about posters who don’t understand the results of their analyses. It promotes “I’ve got some data, what statistical tests can I do on it?”
 

Miner

Forum Moderator
Staff member
Admin
#3
For me, the biggest single issue is that statistical significance (as determined by the p-value) is often detached from practical significance. This has gotten worse with access to big data, where extremely large sample sizes (in some cases, the entire population) will provide a statistically significant outcome that is of no practical importance. The only way to balance statistical and practical significance is to determine practical significance in advance, determine the sample size that will balance statistical significance with this value then run your experiment with that sample size. More often than not, the experimenter just takes the data they have and run with it.

I am not ready to throw out the concept of hypothesis testing because great strides have been made using it. In addition, you can usually run a confirmation experiment in industrial experiments to sort it out. Notice the replication crisis seems limited to soft sciences where they either cannot or will not run confirmation studies and the primary goal is to publish. In industrial statistics the primary goal is to achieve demonstrated improvements, and there are checks and balances in place to prevent wild claims with no basis (e.g., financial verification).
 

John Predmore

Trusted Information Resource
#4
The editorial I read yesterday, on the same subject, was from the American Statistician.
The ASA's Statement on p-Values: Context, Process, and Purpose
https://amstat.tandfonline.com/doi/full/10.1080/00031305.2016.1154108
In summary, "the American Statistical Association (ASA) believes that the scientific community could benefit from a formal statement clarifying several widely agreed upon principles underlying the proper use and interpretation of the p-value. "
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#5
Miner - I understand your position and experience. My experience is a bit different. While the null hypothesis and p value can be helpful if we remember replication, the role homogeneity and practical importance, they aren’t necessarry. And unfortunately too many ‘teachers’ dont’ Talk about replication, homogeneity and practical importance. The recent misuse with big data is just the most recent example. And in my experience the misuse is not limited to the social sciences where planned experiments are difficult. The seduction is that a single small sample with a p value doesnt’ require much ‘work’ or that really hard thing: thinking. Those of us who actually understand it and can use it appropriately are rare.
 

bobdoering

Stop X-bar/R Madness!!
Trusted Information Resource
#6
My first question is usually: What is the distribution? My second one is: What should it be? No statistical calculation can answer the second one - that one takes some thought.
 

optomist1

A Sea of Statistics
Super Moderator
#7
".....helpful if we remember replication, the role homogeneity and practical importance, ...that really hard thing: thinking ", the quote pretty much condenses the matter, at least for me. Statistics Software is an extremely powerful and quick tool....but requires conscious thought
 
Thread starter Similar threads Forum Replies Date
F Does anyone have an ESD quality/cooler talk to share? Training - Internal, External, Online and Distance Learning 4
qualprod Evidence of a talk or phone call in approvals? ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 8
Gman2 Control of Records - Lets talk about modern methods Records and Data - Quality, Legal and Other Evidence 5
AnaMariaVR2 Car Talk Traffic Tickets Funny Stuff - Jokes and Humour 1
Marc Did Stone Age cavemen talk to each other in symbols? Coffee Break and Water Cooler Discussions 7
S Examples of Change and Quality talk at our all hands meeting Misc. Quality Assurance and Business Systems Related Topics 6
S Talk like a Pirate Day - Lets go on a Treasure Hunt Funny Stuff - Jokes and Humour 4
S Fast Talking - People who talk fast Coffee Break and Water Cooler Discussions 24
S Ideas for General Quality Talk Misc. Quality Assurance and Business Systems Related Topics 12
Sidney Vianna Tough talk on Detroit Three doesn't consider suppliers World News 0
Hershal Congrats! Let's talk about top posters in the threads General Measurement Device and Calibration Topics 17
B Service and maintenance company - I have an unusual exclusion to talk about in 7.4 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 23
Marc Pirate Training Needs - Talk Like A Pirate Day - 2006 thru 2011 Funny Stuff - Jokes and Humour 84
A ISO 14001 and ISO 9001 combined? Talk within ISO to combine these two specs ISO 14001:2015 Specific Discussions 12
E Talk about stressing situations on work - Utility of vendor rating for transporter Coffee Break and Water Cooler Discussions 1
M Talk about steel producer consolidation: World News 0
B Can we talk about lunch? Coffee Break and Water Cooler Discussions 27
A Dinner guests and table talk Coffee Break and Water Cooler Discussions 41
C Real Coffee "Break" Talk Coffee Break and Water Cooler Discussions 40
Claes Gefvenberg Talk about expensive mishaps... Nonconformance and Corrective Action 16
Marc Standards Committee Talk ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 0
T Is Chrysler ditching Powerway Customer and Company Specific Requirements 7
S MSA clause 7.1.5.1.1 - Statistical studies on counting weight scale IATF 16949 - Automotive Quality Systems Standard 4
S How to perform verification of the Statistical Analysis Software? Qualification and Validation (including 21 CFR Part 11) 3
M V&V phase: Justification of acceptance criteria (statistical method ) - (Medical Device) Design and Development of Products and Processes 2
Bev D Statistical Alchemy Misc. Quality Assurance and Business Systems Related Topics 1
K Looking for guidance to write an SOP on Statistical Methodologies? Statistical Analysis Tools, Techniques and SPC 7
M Minimum sample size - Guidance and statistical rationale Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
N Design Verification & Process Validation - Statistical sample sizes Design and Development of Products and Processes 2
John Predmore Interactive visualization through graphical simulation of statistical concepts Statistical Analysis Tools, Techniques and SPC 3
Bev D Profound Statistical Concepts Misc. Quality Assurance and Business Systems Related Topics 63
A Statistical Analysis - Check if these organisms at different concentrations affect the growth of wheat seedlings Using Minitab Software 4
H Statistical Techniques Procedure - What should be included Document Control Systems, Procedures, Forms and Templates 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
Marc Steve Prevette's Statistical Process Control (SPC) "Library" Statistical Analysis Tools, Techniques and SPC 0
John Predmore A Balanced view of statistical tests Statistical Analysis Tools, Techniques and SPC 3
V Statistical basis and justification while comparing / changing sampling plans Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
S SPC (Statistical Process Control) for Unilateral Tolerance - Questions Statistical Analysis Tools, Techniques and SPC 6
S IATF 16949 9.1.1.3 Application of statistical concepts - Our technicians are quizzed for statistical knowledge IATF 16949 - Automotive Quality Systems Standard 3
K Please help identify appropriate statistical treatment Statistical Analysis Tools, Techniques and SPC 13
ScottK Statistical basis for 30 pieces for FAI 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7
B IATF 16949 clause 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
A Statistical Process Control and Inspection in Footwear Production Statistical Analysis Tools, Techniques and SPC 0
M IATF 16949 Cl. 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
Steve Prevette Statistical Process Control Library Statistical Analysis Tools, Techniques and SPC 17
Marc Happy Birthday Statistical Steven - 2015 Covegratulations 10
L When are Statistical techniques not applicable? Service Industry Specific Topics 16
M FDA 21 CFR 820.250 - Does "valid statistical" always mean math? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 6
R Common Statistical Errors Using Minitab Software 1
Y Statistical Analysis of Road Traffic Data Statistical Analysis Tools, Techniques and SPC 11

Similar threads

Top Bottom