[Harold]

The worst thing a Japanese would do you to if they don't know you is bump into you accidently in the streets.

The only place you're going to get a big reaction is if you're in a place where you are connected to other Japanese, like school or work. Other than that, the most they'll do to make you feel different is stare at you. Even so, while I was in Shibuya and Akihabara, I got almost no stares, but a billion when at school, if that helps put it better.