I don't own any jeans- would khakis and a white OCBD work?
In Atlanta, bars and restaurants that restrict admission to those over 18 may allow smoking. Do you ever wear your velvet jacket to a place where people smoke? I'd be concerned about it acting like a sponge and requiring a hosing down with Febreze and a trip to the cleaners.
I wouldn't do it with khakis - IMO it would a lot more look out of place than with jeans.
My jacket may smell like smoke for a few hours after being in a bar, but it airs out rather quickly. Certainly by the next time I have occasion to wear it.
As for the comments about not liking jeans and their informality - understood. Keep in mind, though, that not all jeans are grubby work wear. Some very dark, tailored jeans can look quite sharp to many eyes. Cringe if you will, but the uber-successful designer Tom Ford wore a velvet jacket and jeans as a bit of a uniform for some time. Many woman like the look -the key is not too accessorize it so that you come off like a douche - keep the rest of the outfit simple.
A compromise to all of this may be odd wool trousers. I've done this before and it looked good if not stunning. As always, the devil is in the details - the trousers and jacket must complement each other in cut, color and level of formality. Think of how a black velvet jacket is often worn as part of a semi-formal rig; it can look very good if done correctly.
