This is a case where the NB (notified body) looks to be completely wrong, but there are some issues to consider.
It is a common mistake to automatically assume that an object used in a medical application automatically falls under the directive.
The correct interpretation is to consider the labelling manufacturer's intended purpose.
If a doctor buys an ruler from Target's stationary section, and then uses it in her clinic to make diagnostic measurements, the ruler does not become a medical device. But the same ruler bought from a medical catalogue and labelled for medical use is a medical device.
For the battery charger, if it is bought off the shelf, and the original label indicates it is just a battery charger, and all the original labelling remains (including the user manual), then it remains a non-medical device and the notified body has no jurisdiction. The charger falls under the low voltage (LVD) and EMC directive, which I guess it already has been CE marked for.
If it is re-labelled, things get a little messy.
First it becomes a medical device, so various legal matters need to be handled (which are too long to explain here). Really, it is better just to keep the original labelling to avoid these problems.
Assuming it was re-labelled, next is the question of what standard. EN60950 does not appear in the official list of harmonized standards for medical device, so it looks like EN 60601-1 could be the only choice.
There is an argument that EN 60601-1 is not applicable (out of scope) because the charger does not contact the patient. And EN 60601-1 is clearly written for equipment that contacts the patient, much of the standard does not make sense otherwise. In this case, it should be the appropriate EN/IEC standard, even if not harmonized. There is no legal path through which an out of scope standard becomes a harmonized standard by default.
But ... there is also argument that the scope of EN 60601-1 includes "all accessories". This is is a poorly written definition, it is too broad, not workable in practice and conflicts with regulation. It is likely to be changed in the future. While some NBs and test agencies highlight this definition, they do it selectively (when it suits them), not consistently.
Assuming the NB insists on using the "all accessories" definition, for external power supplies (separate power supply) the standard anyhow has an option to use either EN 60601-1 requirements or the appropriate IEC standard, plus the requirements of Clause 16. Since the charger is not used while connected to the patient, Clause 16 is basically not applicable. So the obvious option would be a simple application of IEC 60950.
So no matter what way we go (even using questionable interpretations) we always end at EN 60950.
One caveat: Li-ion batteries are not to be treated lightly. Irrespective of the above discussion the risks of overcharging, short circuit should be considered in the risk management file. Compatibility between the charger and battery is an important part. That does not make the charger a medical device, just part of the input for risk assessment. To assess the risk you may need to access technical information from the charger manufacturer and the battery manufacturer. If you are not qualified to assess this risk, get some outside help.