As others have said, that's between your organization and your Certification Body.
I've experienced one company that rotated every 3 years(ish) in an effort to avoid personal bias. Yes, auditors aren't supposed to have that but they are human (...jury is still out on Randy, however.

).
Bias could go either way. We had one auditor who loved us...absolutely adored us...and we'd sit there, deliberately airing issues and she would not notice them because we were, in her eyes, perfect...at least in comparison to the other sites she went to. It was a compliment to us, but we certainly were not gaining any value from her time with us. Our true value came from our internal audits (my own personal bias is showing here, I suppose

).
We then had another auditor who held us to the requirements of TS16949 instead of ISO 9001 (the standard we were aiming for). We spent more time fighting his nonconformances than we did on actually addressing or resolving them. Perhaps that was his plan...an aggressive passive-aggressive (yes, I meant that) approach to push us towards improvement. Yet again, the value in fighting instead of actually improving was not seen by us.
If you have a good auditor who stays with you, a relationship may develop that could potentially put the auditor in an awkward situation. They are there to audit your processes, but may end up inadvertently consulting by providing possible solutions to discrepancies/opportunities.
The plus side to building a rapport with an auditor is that they see your improvement and growth over the years. They know where you started...they see where you are...and a level of professional trust starts. They will hopefully see a culture form as the organization matures beyond simply "meeting the requirements". Unfortunately, the piece of paper hanging in the lobby is not necessarily about the culture.