It's only mold, you get worse stuff than that on your hands when you go to the toilet to take care of your bodily functions. You don't stop eating with your hands because you use then in the john, you wash them with soap (hopefully) and go on with life.
You could wash them off in warm water and scrub the activated mold spores, clean them with hydrogen peroxide, then non-alcohol mouthwash, again with cold water, and allow them to air dry.
Mold spores are everywhere, they need moisture and the right ambient temp and environment to start growing.