My opinion: don't.
You basically would be doing two things, the first of which is obvioius (and already commented upon) and the second perhaps not so:
- missing the opportunity to be a little bit interesting by varying what you wear;
- marking yourself as "different" from everyone else.
Both seems like a bad idea. The second, in particular, is very bad, unless you are some sort of mad scientist quant, in which case cultivating an air of weirdness could have some benefit. Generally, in the business world (and perhaps more so with high net worth individuals), it's a very bad idea. Rich people can sometimes be eccentric oddballs ... that doesn't mean they want to people who work of them to be the same. Indeed, more likely the opposite.
Even if you stop short of being outright weird, you don't want to be too far out of the norm. The danger is that - instead of being "that guy who knows everything about annuities" or "that guy who's so cool-headed" or whatever it is you want to be - you become "that guy who always wears a red tie."