Maybe, but we are finding this not to be the case. Our customers continue to audit us regardless, so it makes better sense for us not to bother. We maintain 2008 compliance only, and would never consider 2015 version at this point.
I have seen that maintaining ISO registration may or may not reduce customer audits... it all depends on your customer base and how critical your product is to the customer's product. If you are a critical supplier then they will audit regardless, and still require in quality agreements that you maintain some sort of registration via an accredited body.
As far as the OP... I don't know what industry you're in, but I would have a close look at any quality agreements you have with customers before dropping it. I would bet my life savings that if you do business with drug or device customers and you have quality agreements there will be language in them about maintaining registration to a recognized standard.