The number of people isn't nearly as important as the purpose behind the people you talk to.
For example, let's say you're auditing a manufacturing line. You'd probably want to talk to technicians on the manufacturing floor and probably technicians who perform different operations. You'd probably want to talk to some of the folks who maintain the equipment, who developed the processes, who package the product, who conduct quality investigations when things go wrong, management, etc.
So even if there are 100 employees who are in some way part of the manufacturing line, there might only be a half dozen people who work in equipment maintenance. If you just talked to them, you'd get a very different view of the process than if you talked with 6 manufacturing floor technicians. If you talk with some people from each operations, group, and function within the organization, you'll get a much better view than if you just hold yourself to a hard number of people to talk to.