That depends: my reputation with
whom?
Some people actually matter. A lot of those who matter are already people who know you better than to believe any rumors. (That is, friends and family.)
Most people I don't know don't really matter that much to me, and as such they can believe whatever they want for all I care.
The only real problem would be a bad rep with people who don't know you all that well but are still important in some way. Business associates, employers and the like come to mind.