People have opinions about you and talk about you, often. And what they say can be nasty. But here's the thing: what they think about you doesn’t matter. Everyone is allowed their opinion.
What people say about you when you are not around is none of your business. Learn to accept it.
You can't please everyone all of the time, and those you displease will react more strongly than is reasonable. That's how it works.
When you accept that, fear and insecurity will not affect the decisions you make any more and you will become a better co-worker and manager.