Magic number? 2... One(competent one) to do the internal audits, one (competent one) to audit the internal audit process. After that it's going to depend on all kinds of things you didn't tell us about
that's not so bad. you really need enough people to ensure that audits are done by impartial and qualified auditors.
Most important thing - auditees (and auditors too) mustn't take the audits personally
We have 19 internal auditors for QMS for around 750 teammates. I have heard numbers in the past that it is recommended to have ~10% of the workforace as auditors. We also have different auditors for EMS and SMS being environmental and safety. I m trying to get between 25-30 as typically we have some that cannot make it for some reason - travelling to a customer, sickness, etc.
What I do know is that areas that have more auditors naturally do better in audits and those areas have a better understanding of the system. Their audit hat does not just come off when they are not auditing so it filters into the department per se. Areas that have no auditors always seem to struggle more and some of this could be manager committment. This is very consistent even between different companies I have worked for.