How do I know if it is safe not to remove a lesion and just let it be?

This is very difficult to say without seeing the lesion. Your doctor will help you make this decision. Generally if the skin lesion is not growing or changing with time then it is much less likely to be a cancer.