Hello Nancy,
Welcome to the cove! It's a great place to learn and share.
How do you tell if a QS is effective? The list is probably endless. But here is something to get you started.
First, have you identified what your Customers have deemed important? This is always a good place to start. Finding out what makes them happy and relating it to your system creates good business metrics. I should mention though, what your organization 'thinks' is important may in fact, be intirely wrong. What you will need to do is ask your Customers what is important rather than working off internal speculation (perhaps a survey). There is a difference between your Customer's Needs and Wants, so be careful (for a more detailed explanation on Needs and Wants, use the Search option for this bulletin board).
I always recommend balance when creating measurement systems for effectiveness. For instance, in general Scrap and Rework are internally driven measures. Your organization wants to know these figures (caution: some OEM Customers want to know this figure too). From the Customers perspective, availability, choice of colors, on-time delivery may be important. Get some from each category and create measurement and reporting systems. System effectiveness is measured on the output, your organization needs to be concerned about the inputs. If you can, find measures important to both groups and use these when you first begin. Add to the list as your system becomes more stable. Subtract from your list when the data does not produce significant results. Your measures should report on the effectiveness of actions taken on the system to produce a better result. Caution: numbers can be deceiving! Are corrective actions up because the system is failing, or have your auditors improved and detecting more problems because of experience? Know the reasons why things are. Never speculate.
A word of caution from Lloyd Nelson: the most important things in an organization are unknown and unknowable. This is important to remember. No one knows the cost of a lost customer. But the cost is not important here. The lost customer is.
I hope this helps.
Regards,
Kevin