It depends on what and why you are passivating.
Passivation is a cleaning process. Its original intent is to clean free iron from stainless steel as free iron will rust. Free iron can be on the surface of stainless steel for a variety of reasons, including rubbing against ordinary steel.
There are several methods of passivation, with a common one being a nitric acid wash / dip. A less harsh one uses citrus cleaners.
Sometimes passivation is used for other than the original purpose, mainly for sterilization. It is common for some medical equipment to include passivation for sterilization.
Passivation to prepare stainless steel for welding generally would tolerate handling bare handed. Passivation for sterilization would not.