Don't ask me to explain this particular artifact of human behaviour, but I am universally treated better when I'm wearing a blazer, no matter how casual or even ragged my other clothing may be underneath. I have a number of options for a jacket when I'm leaving the house, including a few different hooded sweaters, a fleece windbreaker, a sued jacket and a heavy wool winter coat. The winter coat and the blazers seem to elicit the best reactions out of people. Salespeople smile, bus drivers give a friendly nod, instructors ask how I'm doing, and so on.
But if I wear one of the sweaters? Forget it. Human nature is what it is, I suppose.