How to classify several systems for Quality

[Quality27]

Hello,

I work for a large startup and we are currently in the design phase of a large system. We have identified several levels of systems that are currently in development. As one of the initial steps in Quality, I am trying to classify these systems as 1,2,3,and 4 to help with the level of Quality Assurance activities we need to undertake based on the criticality of the system. Please note that this is a high level classification of systems to provide initial guidance to design, procurement and manufacturing. We will be conducting FMEAs, etc later when we have better clarity on designed components.

How do I go about this criticality classification? Function failure impact, Loss of function during operation, System complexity are some I have on mind but nothing gives me a clear logic that I am looking for.

For example, at the main level, I have 20 systems that make up our final system. Each of these main level systems have 3-5 sub-systems and these sub systems have 5-10 super sub-systems each. If I start with my analysis and a particular main level system has a very higher impact of failure, I classify it as Level 1 within my 1,2,3,4 classification system. Then all the sub-systems and super-subsystems of this main level system get the same classification as any sub-system or super-subsystem failure would cause the main system to fail. Then this becomes a futile exercise for me as the intent of my classification is to put different levels of control based on the criticality of the system. Ideally, my goal within each system is to identify which ones belong to a critical, semi-critical, etc system so that I can build redundancy in design, slect higher end suppliers, verify conformance through inspection and testing so that I have more confidence in the overall functionality of the system we are building.

If anyone has been in similar scenarios or have thoughts to share regarding this predicament I have, I appreciate very much. Suggestions on any different approach are also welcome.

Thanks,

Tony

 

[Steve Prevette]

From my experience with nuclear systems, and also explosives (torpedo maintenance) you generally end up starting from the safety perspective. What is the worst thing that can happen from a safety perspective if something goes wrong. Conventional Explosion? Nuclear Criticality? Injury? Spread of hazardous materials? Spread of radioactive contamination? This starts to dictate the safety and quality controls we put in place, such as reader-worker with signoffs for procedures, versus do the steps in any order you want. And we do put in layers of controls so we don't get to some of the worst case scenarios.

For systems I've dealt with, once we get past the safety issues, then we look at - will the effectiveness / capability of the product be compromised. Issue - the USA went to war in WWII with faulty submarine torpedoes - they did not go "boom" when they hit the enemy ship. Beyond effectiveness, then comes cost effectiveness, preventing budget overruns. Then we tailor the controls as needed.

The US DOE established the Integrated Safety Management System that worked through much of this. See DOE G 450.3-3 TAILORING FOR INTEGRATED SAFETY MANAGEMENT APPLICATIONS

quoting
The Department of Energy (DOE) Integrated Safety Management (ISM) system must support many different kinds of work, from the operation of nuclear and non-nuclear facilities to laboratory experimentation to environmental restoration activities. To accomplish the work safely, and to protect workers, the public, and the environment, the system must function to identify and control all types of hazards, from commonly encountered workplace hazards to rare or one-of-a kind process hazards,

 

[Steve Prevette]

I should also point out from a strictly quality perspective, the classification of minor, major, and critical defects which represents another good risk logic.

• A Minor defect is a discrepancy from the standards, but one that is not likely to affect the usability of an object.
• A Major defect is one that is likely to create failure of the unit for its intended purpose.
• A Critical defect is one that is deemed to be hazardous or unsafe.

From svw5304.tmp (elsmar.com) (MIL STD 105E, here on the COVE).

 

[Quality27]

s, and also explosives (torpedo maintenance) you generally end up starting from the safety perspective. What is the worst thing that can happen from a safety perspective if something

From my experience with nuclear systems, and also explosives (torpedo maintenance) you generally end up starting from the safety perspective. What is the worst thing that can happen from a safety perspective if something goes wrong. Conventional Explosion? Nuclear Criticality? Injury? Spread of hazardous materials? Spread of radioactive contamination? This starts to dictate the safety and quality controls we put in place, such as reader-worker with signoffs for procedures, versus do the steps in any order you want. And we do put in layers of controls so we don't get to some of the worst case scenarios.

For systems I've dealt with, once we get past the safety issues, then we look at - will the effectiveness / capability of the product be compromised. Issue - the USA went to war in WWII with faulty submarine torpedoes - they did not go "boom" when they hit the enemy ship. Beyond effectiveness, then comes cost effectiveness, preventing budget overruns. Then we tailor the controls as needed.

The US DOE established the Integrated Safety Management System that worked through much of this. See DOE G 450.3-3 TAILORING FOR INTEGRATED SAFETY MANAGEMENT APPLICATIONS

quoting
The Department of Energy (DOE) Integrated Safety Management (ISM) system must support many different kinds of work, from the operation of nuclear and non-nuclear facilities to laboratory experimentation to environmental restoration activities. To accomplish the work safely, and to protect workers, the public, and the environment, the system must function to identify and control all types of hazards, from commonly encountered workplace hazards to rare or one-of-a kind process hazards,

Hi Steve,

Thank you very much for the response. I am in a similar situation and with the same line of thinking. But my logic failure is happening when I analyze the functional failure of a main level system, all lower level systems get the same level as they are connected in series to the main level function. This defeats my purpose of having a multi-level QA controls depending upon the importance of a system or sub-system. How do I break this logic? Any thought?

 

[Steve Prevette]

Hi Steve,

Thank you very much for the response. I am in a similar situation and with the same line of thinking. But my logic failure is happening when I analyze the functional failure of a main level system, all lower level systems get the same level as they are connected in series to the main level function. This defeats my purpose of having a multi-level QA controls depending upon the importance of a system or sub-system. How do I break this logic? Any thought?

There is the issue of "Systems Thinking" and looking at how the whole system operates versus dissecting the system. Dr. Russ Ackoff was a great thinker here - he said "You write. Your hand does not write, if you want to prove that chop off your hand and see if it will write". So sometimes, if we dissect the system to the subsystem level - we lose what IS the system and really need to look at the whole.

On the other hand (LOL) certain subsystems may play critical roles. In my example of the bad WWII torpedoes, the exploder pin would shear off if the torpedo hit the target ship at an angle. Before the war, they did little or no testing of the exploder mechanism, and if they did any, it was at a 90 degree angle.

So I would not try to break the logic per se - but analyze "what is the whole system supposed to accomplish?" Of these things/tasks, what subsystems are in play. The motor driving the torpedo was not in play when it came to "did the torpedo explode?". The motor and guidance got the torpedo to the target, but the exploder failed to explode upon contacting the target. Now this required redesign of the exploder, and modern torpedoes actually explode at some depth BELOW the target ship's keel in order to break the keel of the ship. So we really need to know how the components interact in the system in order to assess what risk level they present to safety and mission accomplishment, and mulitple sub systems may contribute. The point is - we really need to understand how the sub systems / components work together to make the system work, and what are the failure modes in the various sub systems and components that contribute to safety / mission failures. Understanding how the WHOLE system works (rather than eating the elephant one bite at a time) gets us to understanding the controls needed.

 

[Quality27]

There is the issue of "Systems Thinking" and looking at how the whole system operates versus dissecting the system. Dr. Russ Ackoff was a great thinker here - he said "You write. Your hand does not write, if you want to prove that chop off your hand and see if it will write". So sometimes, if we dissect the system to the subsystem level - we lose what IS the system and really need to look at the whole.

On the other hand (LOL) certain subsystems may play critical roles. In my example of the bad WWII torpedoes, the exploder pin would shear off if the torpedo hit the target ship at an angle. Before the war, they did little or no testing of the exploder mechanism, and if they did any, it was at a 90 degree angle.

So I would not try to break the logic per se - but analyze "what is the whole system supposed to accomplish?" Of these things/tasks, what subsystems are in play. The motor driving the torpedo was not in play when it came to "did the torpedo explode?". The motor and guidance got the torpedo to the target, but the exploder failed to explode upon contacting the target. Now this required redesign of the exploder, and modern torpedoes actually explode at some depth BELOW the target ship's keel in order to break the keel of the ship. So we really need to know how the components interact in the system in order to assess what risk level they present to safety and mission accomplishment, and mulitple sub systems may contribute. The point is - we really need to understand how the sub systems / componen

Hi Steve,

Thanks again for your patience and helping with this stalemate. I kind of reached the same conclusion as the torpedo pin example and the only stage where we might find it could be while doing a DFMEA with a proper structure and functional analysis. Other than that, I am thinking about the following logic:

Score each identified system in New, Unique, Different, Difficult and Failure Impact (lets say 1-3 score on each of these). If I do this, even though a main level system is higher level and at a sub-system level there is a compressor, it gets a lower score and I may not have to build redundancy or put too many controls on a product that is readily available in the market.

Do you believe this logic could be implemented? Do you see any logical gaps?

 

[Steve Prevette]

Do you believe this logic could be implemented? Do you see any logical gaps?

Seems reasonable to me. A tried and true design should be more reliable and known than something new and innovative. As long as knowledge is applied - for example we don't want to stifle innovation, just be smart about it. And there is looking at the interactions between components - for example there also was a problem with what depth the original WWII torpedoes ran at - the depth control didn't work - and that had some interaction with the torpedo hitting the ship rather than being set off by the magnetic detonator.

Something to consider is the testing program for verification of these risks and assumptions once initial prototypes or even sub assemblies are complete.

 

[BradM]

Hello there!

You mentioned these classifications are to assess what approach is to be undertaken within the quality system. Correct?

So I would recommend identifying the 1-4 classifications system based on quality failure.

So say it's a... restaurant, with 1 being minor and 4 being major...

1. Will this failure lead to a potential service quality near miss? Will this failure lead to management intervention?
2. Will this failure lead to a service failure? Will this lead to rejected finish product and a customer dissatisfaction?
3. Will this failure lead to a broader systemic failure? Will multiple orders be rejected and lead to a potential drop in sales?
4. Will this failure lead to a large loss of raw goods? Will this failure potentially lead to customers receiving poor quality product?

I just kind of threw that together, so you can probably see holes fairly quickly.

You could also tie it to (since you mentioned design) the cost of quality phases:
1- Prevention costs
2-Inspection costs
3- Internal quality failure costs
4- external quality failure costs.

 

[Quality27]

Do you believe this logic could be implemented? Do you see any logical gaps?

Seems reasonable to me. A tried and true design should be more reliable and known than something new and innovative. As long as knowledge is applied - for example we don't want to stifle innovation, just be smart about it. And there is looking at the interactions between components - for example there also was a problem with what depth the original WWII torpedoes ran at - the depth control didn't work - and that had some interaction with the torpedo hitting the ship rather than being set off by the magnetic detonator.

Something to consider is the testing program for verification of these risks and assumptions once initial prototypes or even sub assemblies are complete.

Thanks Steve, great to share thoughts with you. I believe this can be implemented with extensive verification/validation testing for very high risk sub systems.

I will update you on how this thinking develops.

 

[Quality27]

Hello there!

You mentioned these classifications are to assess what approach is to be undertaken within the quality system. Correct?

So I would recommend identifying the 1-4 classifications system based on quality failure.

So say it's a... restaurant, with 1 being minor and 4 being major...

1. Will this failure lead to a potential service quality near miss? Will this failure lead to management intervention?
2. Will this failure lead to a service failure? Will this lead to rejected finish product and a customer dissatisfaction?
3. Will this failure lead to a broader systemic failure? Will multiple orders be rejected and lead to a potential drop in sales?
4. Will this failure lead to a large loss of raw goods? Will this failure potentially lead to customers receiving poor quality product?

I just kind of threw that together, so you can probably see holes fairly quickly.

You could also tie it to (since you mentioned design) the cost of quality phases:
1- Prevention costs
2-Inspection costs
3- Internal quality failure costs
4- external quality failure costs.

Thank you for your thoughts. I am trying to classify systems, not defects. I will definitely assign these ratings to the nonconforming scenarios.