I'm going to throw in with the auditor. I think the others have said nothing because you've been doing what they expect. This one may be on to something!
Simply put, telling the rest of management which clause numbers etc are part of the audit findings is not much use, frankly. They have no time for what ISO says! Now, the other CB auditors all live in that world! So, when you speak their language, why would they be concerned? I'm guessing you took an internal or Lead Auditor course and that's what they teach you - but that's not appropriate for internal audits, IMHO. Better to relate it to their process, performance, etc.
I too, like the 'quick and dirty' approach to audits. Notes made on paper, of your own observations, samples etc are far more effective than reams or hours spent in typing copious pages of report. You need to make it painless for yourself, management need the 'elevator pitch' in a report.
Despite what the wrongs and rights of the way the auditor did this, for a minute - ask yourself this: Do management understand what I'm telling them? Really, really?
As Roxane says, he didn't hammer you, it's just that he had a lot of different ideas about what you've been told - and what you've been led to believe by the others' silence - is a good way to audit. Actually, if I were the auditor, I'd probably say something similar!