Profound Statistical Concepts

Profound Statistical Concepts 2019-10-31

Bev D

Heretical Statistician
Staff member
Super Moderator
#31
I prefer just "finding out," rather than proving right or wrong, although the latter is a better semantic fit with the notion of rejecting a hypothesis. I have the annoying habit of jumping in when someone says they are doing a test or study "to show that..." and saying, "No, we are doing that "to find out if..."
Well, since I am opposed to the null/alternative hypothesis approach we agree.
‘Pure’ science as you describe still requires discovery of underlying causal systems and/or outputs. In this case it’s no different really than problem solving or product development. I really tire of the argument that ‘my thing’ is different, better, unique than ‘your thing’ competition.


I’m still not sure of your point?
 
Elsmar Forum Sponsor

Jim Wynne

Staff member
Admin
#32
So when an experiment is designed to show a certain expected outcome, and that outcome actually comes about, the experiment is highly inefficient in terms of gathering new knowledge.
If an experiment is designed such that the designer of the experiment already knows the outcome, it's not an experiment--it's a demonstration.
 

Bev D

Heretical Statistician
Staff member
Super Moderator
#33
So probability and statistics is a complex science. Casual understanding is not understanding it's acquaintance.
to paraphrase the tagline that one of our cove members uses: "words have different meanings"

"Experiment" is one of those words - I prefer study design but that doesn't necessarily help here. Study designs or experiments have different purposes:

The enumerative study seeks to estimate parameters of a population in order to determine what to do with it. Inspection census and surveys are all examples of enumerative studies. of course we learn something from these studies, but we only learn about the specific population under study. we cannot make predictions from these studies.
The Analytic class of studies is about understanding the underlying system to some useful level in order to predict future behavior. Analytic studies fall in to 3 broad purposes (maybe more?)
  1. Exploratory: typically looking fro rough patterns and extent. it describes what's happening in high level terms.
  2. Diagnostic: fine detail of the causal mechanism and/or what will improve it.
  3. Confirmatory: verification and validation that we got it right.
Although there is - hopefully - no new 'learning' from a confirmatory study, there is still immense value in demonstrating that we got it right. Many Customers and our own internal QMS require this demonstration. in automotive the demonstration takes the form of capability studies. in medical devices they take the form or IQ, OQ and PQ validations. The FDA, FAA and USDA all require some level of confirmatory study to acquire licenses to sell product. and the study design that demonstrates the validity of our 'claim' is also critical. I've known many people who biased their study to show a result they wanted to 'prove'. This is scientific malpractice at best and illegal at worst.
 

Watchcat

Trusted Information Resource
#34
it's not an experiment--it's a demonstration.
Indeed, "demonstrate" is frequently used in both pharma and medical devices. Regulators use it, meaning you have to demonstrate it to them. But some developers also use it, because they assume it is a foregone conclusion. From a meeting to decide how long patients should be followed in a clinical trial in order to adequately assess safety and effectiveness of a particular type of medical device:

"Some people think 12 months of data are needed. Some people think at least 6 months. Others think only 3 months will be sufficient. And some people seem to think they already know it is safe and effective, without ever having collected any data at all."
 

Watchcat

Trusted Information Resource
#36
when an experiment is designed to show a certain expected outcome, and that outcome actually comes about, the experiment is highly inefficient in terms of gathering new knowledge.
Point well taken, but expectations are not knowledge.
 

Watchcat

Trusted Information Resource
#37
the death of the p-value
Is it dead in scientific journals, which is where this discussion started? I don't read a lot of scientific journals these days, but still some. I read a lot more medical journals. In the journals I read, the p-value seems to be very much alive and well. If you call being reported in a medical journal living. :p
 

Ed Panek

QA RA Small Med Dev Company
Trusted Information Resource
#38
I think the situation is similar in "science."

Because there are multiple levels of definitions of "science," people with very diverse levels of education and intellectual development can legitimately call themselves "scientists," and many do. Unfortunately, many of these "scientists" feel the need to conduct scientific research, even though they may lack the education and/or the intellect to do so. They are a "scientist" and therefore it seems that any research they do is, by definition, "scientific" research. In many cases, I'm not sure an understanding of statistics and experimental design is even expected.
We had a similar review with IDE review at the FDA. "Your file is being reviewed by Dr John Doe, PhD" Google search reveals his PhD is in electrical engineering. I tell the team - expect a ton of electrical questions. But only 5% of this device is about electronics. It wont matter, trust me. Your file is now being reviewed by Sarah Doe PhD Google search their background...etc
 

Watchcat

Trusted Information Resource
#39
Ed, don't get me started. Really. Some day I will put together and post a carefully crafted and extensive rant, but today is not that day. For today, just don't ever use "Regulatory S-word" in my (physical or virtual) presence. For both our sakes, please.
 

Ronen E

Problem Solver
Staff member
Moderator
#40
If an experiment is designed such that the designer of the experiment already knows the outcome, it's not an experiment--it's a demonstration.
There's a difference between knowing the outcome and hoping for a specific one. Say that the (unknown upfront) probability of the hoped-for outcome is 80%* - there'd still be some accumulation of new knowledge in running the experiment, but it'd be much less than when that probability is, say, 50%. It may look obvious, but surprisingly lots and lots of experiments resembling the former take place.

*) One could argue that this would be "a demonstration with a 20% chance of failing", but for me the difference between that and "an experiment" falls in the wordsmithing department.
 
Thread starter Similar threads Forum Replies Date
W Deming's SoPK (System of Profound Knowledge) Discussion Philosophy, Gurus, Innovation and Evolution 220
W Deming's SoPK (System of Profound Knowledge) Challenge Philosophy, Gurus, Innovation and Evolution 66
S How to perform verification of the Statistical Analysis Software? Qualification and Validation (including 21 CFR Part 11) 2
M V&V phase: Justification of acceptance criteria (statistical method ) - (Medical Device) Design and Development of Products and Processes 2
Bev D Statistical Alchemy Misc. Quality Assurance and Business Systems Related Topics 1
K Looking for guidance to write an SOP on Statistical Methodologies? Statistical Analysis Tools, Techniques and SPC 7
M Minimum sample size - Guidance and statistical rationale Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
N Design Verification & Process Validation - Statistical sample sizes Design and Development of Products and Processes 2
John Predmore Interactive visualization through graphical simulation of statistical concepts Statistical Analysis Tools, Techniques and SPC 3
A Statistical Analysis - Check if these organisms at different concentrations affect the growth of wheat seedlings Using Minitab Software 4
H Statistical Techniques Procedure - What should be included Document Control Systems, Procedures, Forms and Templates 4
O Statistical justification of sampling size in V&V tests ISO 13485:2016 - Medical Device Quality Management Systems 5
optomist1 It’s time to talk about ditching statistical significance Statistical Analysis Tools, Techniques and SPC 6
Marc Steve Prevette's Statistical Process Control (SPC) "Library" Statistical Analysis Tools, Techniques and SPC 0
John Predmore A Balanced view of statistical tests Statistical Analysis Tools, Techniques and SPC 3
V Statistical basis and justification while comparing / changing sampling plans Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 11
S SPC (Statistical Process Control) for Unilateral Tolerance - Questions Statistical Analysis Tools, Techniques and SPC 6
S IATF 16949 9.1.1.3 Application of statistical concepts - Our technicians are quizzed for statistical knowledge IATF 16949 - Automotive Quality Systems Standard 3
K Please help identify appropriate statistical treatment Statistical Analysis Tools, Techniques and SPC 13
ScottK Statistical basis for 30 pieces for FAI 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 7
B IATF 16949 clause 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
A Statistical Process Control and Inspection in Footwear Production Statistical Analysis Tools, Techniques and SPC 0
M IATF 16949 Cl. 7.1.5.1.1 - Statistical studies shall be conducted IATF 16949 - Automotive Quality Systems Standard 3
Steve Prevette Statistical Process Control Library Statistical Analysis Tools, Techniques and SPC 17
Marc Happy Birthday Statistical Steven - 2015 Covegratulations 10
L When are Statistical techniques not applicable? Service Industry Specific Topics 16
M FDA 21 CFR 820.250 - Does "valid statistical" always mean math? 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 6
R Common Statistical Errors Using Minitab Software 1
Y Statistical Analysis of Road Traffic Data Statistical Analysis Tools, Techniques and SPC 11
B Class II Medical Device Manufacturer - SOP for 820.250 Statistical 21 CFR Part 820 - US FDA Quality System Regulations (QSR) 3
E Correct Statistical Test comparing 2 Groups Statistical Analysis Tools, Techniques and SPC 14
A Statistical Software Calibration using Ford's "Sample Calibration File" Statistical Analysis Tools, Techniques and SPC 8
J Defining Martial Arts and Gymnastics Statistical Techniques Statistical Analysis Tools, Techniques and SPC 4
J Capability Analysis - Unusual Statistical Distribution of my Proccess Capability, Accuracy and Stability - Processes, Machines, etc. 5
M PhD Thesis Data Statistical Analysis Methods Statistical Analysis Tools, Techniques and SPC 2
J Statistical Significance and SPC Control Chart Reports Statistical Analysis Tools, Techniques and SPC 9
N Statistical Quality Improvement Action for Small Batch Production Statistical Analysis Tools, Techniques and SPC 17
V Validation of macro - scripts - programs used in statistical software (Minitab-SAS... Qualification and Validation (including 21 CFR Part 11) 5
Moncia Statistical Process Control Crash Course - Question Quality Manager and Management Related Issues 10
H Statistical Models for Predictive Management of Software Processes Software Quality Assurance 2
A Statistical Correlation between ordered SKUs Statistical Analysis Tools, Techniques and SPC 8
S Minitab and Crystal Ball Statistical Analysis Software Using Minitab Software 13
M Determining if two different X's have any Statistical Significance on the Y's Statistical Analysis Tools, Techniques and SPC 4
I Statistical Stability for the PQ of Analytical Equipment Qualification and Validation (including 21 CFR Part 11) 1
F Statistical Comparison of Product: High Average vs. Low Range Capability, Accuracy and Stability - Processes, Machines, etc. 13
E Using ANOVA during the PQ Validation Run to evaluate Statistical Differences Statistical Analysis Tools, Techniques and SPC 4
W Gage R&R for gage pins used to inspect a hole ID called a Statistical Tolerance Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 3
B AS 9100C sec 8.2.4 - " Recognized Statistical Principles" meaning AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 7
R What is PSW - Statistical Process Package + Level 5 APQP and PPAP 7
O Is SPC (Statistical Process Control) always required? Statistical Analysis Tools, Techniques and SPC 4

Similar threads

Top Bottom