Your nonconformance/postmarket/recall process should guide you here - it should require you to do some sort of assessment (Health Hazard Evaluation, Health Risk Assessment, etc) to assess the risk associated with the defect and determine whether a recall is necessary. All that should likely be linked to the documentation for the nonconformity and corrective action.
The webpage below has some general info about recalls and the considerations you should take, but this will all depend on what your internal processes say.
Info related to recall of medical devices, corrections and removals; regulations and Federal Register notices.
www.fda.gov
If you cannot trace any real health risks to the missing label, then you may be able to justify why you don't need to do a recall. But you need to make sure you are following your own procedures and documenting appropriately.