Reality like this shows you who your true friends are. If they are talking about you behind your back then the are not true friends. If anything they should at least try to help you. The one that never abandons you in the heat of the battle is your true friend.
I'm reminded of a story from FFIX. Where Ipsen went to a different city to find a job and support his family. At his job he had a friend named Collin. One rainy day, Ipsen got a letter that was so wet the only words he could read were "come home". So he left and Collin went with him. During their journey they fought all kinds of monsters. One day, Ipsen turned to Collin and asked him "Why did you come with me?" Collin replied "only cause I wanted to."
A true friend would have your back no matter what. If they are not willing to support you then let them go.
Of course this is just my opinion. You don't have to take my advice.