Large data sets of continuous individual data - Estimated or actual deviation

P

pavelstat2019

#1
Hi,

if we have large data sets of continuous individual data e.g. above > 500 value, should will still use estimated deviation or we can use normal deviation over the whole data to calculate UCL and LCL?

UCL / LCL based on estimated deviation (CL+/-2.66MR) in our case are more conservative than actual deviation over general population.

Thanks.
 
Elsmar Forum Sponsor

Bev D

Heretical Statistician
Staff member
Super Moderator
#2
you have posted this same question in an old thread - here is my response to that:

the phrase 'estimated standard deviation' is a misnomer. you must still calculate the control limits using the within subgroup standard deviation. this is how SPC works.

If the process is homogenous the within subgroup standard deviation adjusted by d2 or c4 will be a good estimate of the population standard deviation. (hence the term 'estimate' but using that term instead of within subgroup SD leads to many mis-interpretations.). IF the process is homogenous the subgroup averages will vary by the average +/- 3(within SD/c4)/square root of n. If the process is not homogenous the averages will not comply with the calculated limits. that is how SPC works. trying to calculate the population SD directly sidesteps this critical part of the process.

I am curious tho. what do you mean by having large data sets with n>500? are the data sets your 'subgroups'? or are they for different processes? can you describe what you are trying to do?
 
P

pavelstat2019

#3
We are developing helpdesk dashboard that is based on analytical system that can easily process millions of records in seconds using any custom logic and dashboard visualization is so advanced that 300 days can be shown visually with no problem.

We have a number of metrics like "number of incidents" with historical data for 300 days. We can group those values in weeks / month however that will rather create a problem - day view is much better for operational purposes. "Number of incidents" is also obviously exist within minute / hour but on that scale this metric do not mean anything.

We want to create a control chart that in live mode day-by-day shows whether helpdesk is in control for the last 90 days to signal executives whether their reaction is required. Previously, there were situations in which information about "disaster" did not reach higher level in time.

We analyzed historical data (300 days) using standard formulas with CL = average, UCL / LCL = CL +/- 2.66 * MR and found several periodic peaks due some mismanagement causes however standard deviation calculated using normal formulas SQRT ( SUM (X-AVG(x))^2/(N-1))) it is slightly different from (Average Moving Range) / d2 with d2 = 1.128 for Range of span = 2. This difference is sufficient to exclude some of the peaks.

So, the question we have:
1. Can we use normal standard deviation for historical data and estimated one for 90 days view?​
2. Whether we should exclude from analysis the peaks that do not breach UCL/LCL based on normal formula of deviation (SQRT ( SUM (X-AVG(x))^2/(N-1)))) ?​
3. Going forward, shall we use UCL / LCL = CL +/- 2.66 * MR or we can use UCL / LCL = CL +/-3*SQRT ( SUM (X-AVG(x))^2/(N-1))) for 90 days view ?​
To get some idea about numbers.
CL = 67, MR = 23

UCL1 = CL + 2.66 MR = 129, est. standard deviation = 21
UCL2 = CL + 3 * SQRT ( SUM (X-AVG(x))^2/(N-1))) = 157, actual standard deviation = 30
 
Last edited by a moderator:
Thread starter Similar threads Forum Replies Date
M Leaving Footprints in the Data - Large Database Validation Statistical Analysis Tools, Techniques and SPC 7
V Sample Size for Attribute Data - Large Production Lots Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
supadrai Remote Opportunity - Large Scale Glove Factory QMS / RA / Legal Job Openings, Consulting and Employment Opportunities 0
A Earth leakage current test for Permanently installed Large scale MEE IEC 60601 - Medical Electrical Equipment Safety Standards Series 5
A Large outside micrometers calibrations General Measurement Device and Calibration Topics 2
D Gage R&R Study on Load Cells - Large chemical blending tanks IATF 16949 - Automotive Quality Systems Standard 1
M Informational EU – Minutes of the 24 July 2019 SCHEER Working Group on safety of breast implants in relation to anaplastic large cell lymphoma (BIA-ALCL) meeting Medical Device and FDA Regulations and Standards News 0
M Gage R&R with large inter part variance - Torque Gage R&R (GR&R) and MSA (Measurement Systems Analysis) 7
S Gage R & R for large amount of inspections IATF 16949 - Automotive Quality Systems Standard 10
M FDA News USFDA – Breast Implant Associated- Anaplastic Large Cell Lymphoma (BIA-ALCL) – Letter to Health Care Providers Medical Device and FDA Regulations and Standards News 0
S My department inherited a large number CAPAs ISO 13485:2016 - Medical Device Quality Management Systems 4
M Tracking External Oversight at a large Naval Shipyard Quality Manager and Management Related Issues 9
K Testing Large Plastic Pipe Imported Legacy Blogs 0
L Large sand steel casting discontinuities Manufacturing and Related Processes 8
J Large vs. Small Quality Manual - ISO 9001:2008 Clause 4.2.2 Quality Management System (QMS) Manuals 19
P IT tools for Knowledge Sharing in a large Organization Misc. Quality Assurance and Business Systems Related Topics 1
T FAI (First Article Inspection) of Large Assemblies/Tooling after move to new building AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 12
M David vs Goliath (Large Suppliers vs Low Volume Purchasers) Supplier Quality Assurance and other Supplier Issues 11
R Which large corporations require their suppliers to be ISO 9001 certified? Registrars and Notified Bodies 14
J Thread Gauge Calibration Failure for Pitch Diameter being too large General Measurement Device and Calibration Topics 3
M Large Construction Project Quality Plan - ISO 9001-2008 using ISO 10006 ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 18
M Large Sample Size - up to 32 - for Control Charts Capability, Accuracy and Stability - Processes, Machines, etc. 2
N Definition of Large-Scale Fixed Installation per the RoHS2 Directive EU Medical Device Regulations 3
D Single-Point Scale Calibration - Very Large Scales General Measurement Device and Calibration Topics 3
D Large, untoleranced radius of 20? - Inspection and Tolerancing Inspection, Prints (Drawings), Testing, Sampling and Related Topics 3
R Recently purchased a large number of gages (measurement equipment) General Measurement Device and Calibration Topics 13
T How to move from a small company to a large company Career and Occupation Discussions 7
Proud Liberal Minitab - Organizing a large number of graphs using file details Using Minitab Software 5
J ISO13485 File-Index getting too large Document Control Systems, Procedures, Forms and Templates 2
H Certification of a large commercial helicopter acc. AC29-2C / CS29 EASA and JAA Aviation Standards and Requirements 1
M What is an acceptable PPM for large volume fasteners? APQP and PPAP 6
Q "Enterprise Certification" in Large Companies ISO 9000, ISO 9001, and ISO 9004 Quality Management Systems Standards 3
K Using a Tape Measure for +/- 1/16" Tolerance - Large Welded Assemblies (6 to 12') Inspection, Prints (Drawings), Testing, Sampling and Related Topics 12
B Capability Analysis of a Small Sample Size (as small as 2 and as large as 30) Statistical Analysis Tools, Techniques and SPC 14
D "On Time Delivery" Flowdown from Large Corporation AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 6
L DPPM (Defective Parts Per Million) Reporting - Large Quantity Approach Benchmarking 8
A The use of SPC in Graphic Design and Large Printers Lean in Manufacturing and Service Industries 14
D CMM vs. Measuring Arm for both Large and Small Items Inspection, Prints (Drawings), Testing, Sampling and Related Topics 26
S Special Inspection Levels - Batch sizes are large (typically 100k - 150k) AQL - Acceptable Quality Level 10
A What is a good Sampling Plan for LARGE Lots of Supplier Components Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
S AS9100 REV B Sampling Requirements - Heat Treatment of Large Cylinders AS9100, IAQG, NADCAP and Aerospace related Standards and Requirements 2
bobdoering Dr Wheeler pokes at the Theorem of Large Numbers Statistical Analysis Tools, Techniques and SPC 2
F Determining the Number of Samples for receiving large lots of wire? Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2
O Average Life Cycle Duration for a large number of medical equipment types Other Medical Device and Orthopedic Related Topics 2
C Seeking a holder/jig to calibrate large OD micrometer calipers (up to 23") General Measurement Device and Calibration Topics 8
J Large Weldment Inspection Plan Inspection, Prints (Drawings), Testing, Sampling and Related Topics 6
J Large Projects, large number of partners? Document Numbering/Identifying Issues Benchmarking 6
K Inspection filter needed - Large machine tools Inspection, Prints (Drawings), Testing, Sampling and Related Topics 5
A How to treat large Manufacturing Equipment providers - Opinions and comments wanted Supplier Quality Assurance and other Supplier Issues 5
Proud Liberal Using Multiple Plug Gages check a large diameter Inspection, Prints (Drawings), Testing, Sampling and Related Topics 2

Similar threads

Top Bottom