It won't be traditionally "correct" but if it works for you, knock yourself out.
My biggest hesitation re: doing this wouldn't be the "rules", but rather the risk of it looking too much like the sort of black tie gear Steven Seagal sometimes wore in the mid/late 90s... not exactly one of my style role models.