The folks from my company told me that its not worth the fight..So I think I won't win on this one. I would think that it would be more difficult next time, since I tried to challenge him during this last audit.
Without you making a formal issue with your CB, the auditor feels validated with their comment. And so, they bring their biases to the next client and so on. So, it may not be a 'big deal' at an individual client level, but over the course of many audits, these (and similar) biases are manifested time and again. It is the Voice of the Customer on which we rely so heavily to improve the credibility of services. Too many people like to 'bash' CBs and their auditors, or poorly performing supplier who have an ISO certificate, but they are unwilling to give feedback to help be part of the improvement!
Sadly, when -or even if - the auditor is 'witness' audited, their biases may not surface (maybe it's a different industry etc). So, for those of us in the CB services and all those clients and consultants on whom the success of Certification is reliant, I'd ask you to - at least - write and inform your CB management of your dissatisfaction with this particular finding and that you'd like them to inform you how they have identified such biases so that other auditors won't bring them to your audits again!

I'm reminded of the probably apocryphal story about Ty Cobb, who long after his retirement from baseball was allegedly asked by a writer how he thought he would fare against modern pitching. He said, "Oh, I think I'd probably be lucky to hit .270 against today's pitchers." The writer said, "Why is that? Are today's pitchers really that much better?" "No," said Cobb, "but I'm seventy-five years old."