You have just described 99% of US businesses, no? How often do you have your audits? Set up the next audit during current one, and make sure that management agrees. There is no law that says everyone has to be there during an audit. (you should attempt to get the people leading departments to be audited) If a key person is not going to be there (cannot change his/her vacation) just bring in someone to cover for them, but be able to prove that this sub is competent in the area. There is not one place in the standard that says every person employeed at a registered company has to be present during an audit.
This is not the end of the world. Now, if the same people are gone every single time the auditor comes in, the auditor may begin to smell a rat.