Tests for contingency plans

[Bev D]

So first a little story to illustrate my points. About a decade or so ago we had a huge rainstorm. Our Customer Service call center was in a separate building with only one entrance to the parking lot. Emergency response officials ordered us to evacuate the building as the entrance went over a culvert that was overrun and a nearby river was about to crest and there was 3 inches of water on the entrance expected to rise by several feet. Since we didn’t want to shut down our call center (which serviced Vets all over the US) we marshaled our IT department to put up PCs/laptops and the special call center phone lines in our main building - while everyone was still working. It took 2+ hours of panic and despair to get the center up and running so the evacuation could occur before people were stuck. It was horrible and no one wanted to go thru that again. we built a second entrance uphill and away from the first entrance. AND IT/customer service developed an emergency plan. Then in 2020 our IT department belatedly woke up to the coming shutdown of the pandemic and ordered that all but front line essential workers and their direct supervisors were to work from home on a Friday to ‘test’ the IT system for a full on work from home situation. They then would have a weekend (!?) to fix an gaps or problems. Fortunately the system held as teh state shutdown effective that following Monday. The problem was that many people had left essential things at work thinking they would return on Monday; also many people simply didn’t have the ‘office equipment’ to work form home for any extended period of time and we ended up buying a ton of office equipment for people. We also had a crash course in Microsoft teams and new protocols to develop…

SO: Think about it, plan for it and test it where it is possible. It’s like a. Fire drill. Yes you may have to stop production for awhile (we did it every year) but you can plan for that. All it takes is a little creativity and maybe asking very specific questions here instead of general ones. Remember too that some things can’t be physically tested - there will need to be a ‘thought experiment’ for those few things. Of course some things will not be thought of (who would have thought there might be a pandemic? That’s sarcasm by the way) And regardless of any requirement for a record good business sense tells you that you should have report of the test and what needs to be fixed/improved…or else the test is just a silly check the box exercise…

Although I’m not sure this falls into the QMS in a lead role except for the things that directly effect part quality like a chemical/fabrication process that is interrupted mid-process and what the disposition of hte material would be…

 

[Golfman25]

Lets face it. As Mike Tyson says -- everyone has a plan until they get punched in the face. We can have great plans, tested and whatever. But when the real thing happens, it's like a knuckle curve ball -- have to be able to think and adjust on your feet. For most of us, it's impossible to plan and/or have redundancy for every situation.

 

[Bev D]

Well Tyson - if he said it - was variation on the military saying that plans are great until the first shot is fired. And that is true which is why ‘practicing’ helps people remember what to do, know what works and what doesn’t, etc. Whiel we certainly can’t anticipate everything there are many things we can and we can prepare for them: I’ve never lived in a state that didn’t mandate annual fire drills. They were always helpful. If you are located in a hurricane zone you would be pretty remiss if you didn’t respect that and didn’t have a plan for shutting down and securing vital equipment and preparing the safety of your employees and buildings and inventory, etc. there is no reason to run around like a decapitated chicken for these types of things. But again I’m not sure they belong in the QMS primary role…

 

[Bev D]

Oh yeah and most of my IT groups always had a ‘roll back’ plan and the groups getting an update would have a ‘do it by hand’ plan to maintain operations should a software upgrade go belly up. Needed every couple of years. Betcha lots of people wish Microsoft/CrowdStrike had one of those.

 

[Golfman25]

Oh yeah and most of my IT groups always had a ‘roll back’ plan and the groups getting an update would have a ‘do it by hand’ plan to maintain operations should a software upgrade go belly up. Needed every couple of years. Betcha lots of people wish Microsoft/CrowdStrike had one of those.

Apparently they do -- same thing, 1000s of computers "updated" overnight have to be rolled back by hand.

Why I never do automatic updates. Don't trust them.

 

[Ed Panek]

In the military we trained constantly. Even though a fire board ship is remotely likely most of the time, any fire becomes a threat to the ships survival. We all trained to gear up and put out fires. All hands on deck.

I was on the ships security team so we weekly had security drills with a made up role played intruder. It was fun but mostly the things you need to do when it’s real are muscle memory. Like learning to play a musical instrument. After time you don’t think about it you just play.

The attempt on trump recently shows the power of training. There were no discussion or debate. Secret service They acted and secured the area. Granted how the attacker seemed to be so easily detected yet undeterred is a concern.

 

[Golfman25]

In the military we trained constantly. Even though a fire board ship is remotely likely most of the time, any fire becomes a threat to the ships survival. We all trained to gear up and put out fires. All hands on deck.

I was on the ships security team so we weekly had security drills with a made up role played intruder. It was fun but mostly the things you need to do when it’s real are muscle memory. Like learning to play a musical instrument. After time you don’t think about it you just play.

The attempt on trump recently shows the power of training. There were no discussion or debate. Secret service They acted and secured the area. Granted how the attacker seemed to be so easily detected yet undeterred is a concern.

Wow. Did you really watch any video on that attempt and compare with the Reagan video. With all due respect to the agents, not even close.

 

[Bev D]

Did you watch the Reagan video? the secret service did nothing about the shooter either and he was only a few feet away (well within their security perimeter) but he got off several shots, severely wounding 3 other people* as well as Reagan who damn near died.

Please re-read Eds post: he is commenting on the secret service response after the shots were fired. That was his point…and he did point out the weakness of actual plan that involved the interface between the local police and the secret service…he never mentioned Reagan, or JFK or Ford…

*Press Secretary James Brady, Secret Service Agent Timothy McCarthy, and policeman Thomas Delahanty

Assassination attempts are horrible, difficult to plan for prevention and are the epitome of ‘getting punched in the face’ despite good plans and reacting appropriately.

 

[Ed Panek]

Wow. Did you really watch any video on that attempt and compare with the Reagan video. With all due respect to the agents, not even close.

Yes. Now imagine no training for secret service. What would that look like? Answer lots of dead leaders. That’s this threads message. Training matters. Even a poor showing doesn’t imply “get rid of training. It’s useless for secret service”


What should happen is a lessons learned and updates to security training.

If you want to criticize secret service that’s fine but once the potus was injured within a few seconds he was protected by bodies to intercept bullets. As an agent it takes about a second to understand what’s happening. Then a few seconds to reach potus. The main issue for me is the apparent detail of the shooter being identified and contained. That was bad but the immediate containment of securing potus I thought was about average but really how many samples do we have to compare?

 

[Golfman25]

Yes. Now imagine no training for secret service. What would that look like? Answer lots of dead leaders. That’s this threads message. Training matters. Even a poor showing doesn’t imply “get rid of training. It’s useless for secret service”


What should happen is a lessons learned and updates to security training.

If you want to criticize secret service that’s fine but once the potus was injured within a few seconds he was protected by bodies to intercept bullets. As an agent it takes about a second to understand what’s happening. Then a few seconds to reach potus. The main issue for me is the apparent detail of the shooter being identified and contained. That was bad but the immediate containment of securing potus I thought was about average but really how many samples do we have to compare?

I suppose if you think it’s “well trained” to pick up glasses, hats, and leave the protectee wide open for a secondary shooter, while escorting him to the vehicle, then you’re right. Not wanting to make this a security thread, had the SS done what IATF requires and tested their plan they may have realized using undersized agents to protect an oversized man isn’t going to work that well. That is precisely what the IATF requirement is for — to see and address the potential holes in your contingency plan.