When they talk about the "legal manufacturer address", they are referring to whomever the legal manufacturer is. This may be different from the actual manufacturer.
I argued the same clause from EN1041 to our Notified Body - "The full postal address may not be necessary provided that the address is of sufficient detail that the manufacturer, and if the device is imported into the community, the person responsible, the authorized representative, or the importer can be contacted, e.g., name or trade name, post code and country" - as justification for only having the P.O. Box number on the label. My argument was unsuccessful. The reasoning being that the section of EN1041 that is in is informative.