You give people an inch and they'll take a mile and all your stuff and probably the clothes off your back.

Only way to make sure they don't mess with you is to advertise that you're a guy not to be messed with. If some of 'em hear that you're willing to back off, what do you think they're gonna tell everyone else?