Well, I don't know. Today I didn't have on a vest when the subject comments were made as I was wearing a double-breasted suit from Burberry's. But, virtually all of my single-breasted tweed and other wool suits have contrasting vests and if I have a look, I suppose that's it.
Maybe I should also mention that when I do see others around here in coats and ties I rarely see them also wearing overcoats on cold days, raincoats when it rains or hats or boots on either occasion. I do wear fedoras and homburgs, overcoats, raincoats and protective footwear when appropriate. Perhaps that all combines to appear over dressed to some. I suppose your guess is as good as mine...
Hans