Let's pretend for a moment that we aren't all terrified of clothing.
Me? I'd just wear them... meaning as though they're nothing special. Same rules as jeans - wear them with a nice v-neck merino wool sweater, or a snazzy button-down shirt (assuming the trousers are a solid color). It's fun to just go ahead and invoke the Seventies, and pair them with Chelsea boots or the like. A nice Balmoral boot, just as good. Fancy loafers/driving shoes too.
Obviously, you can wear them to parties, but NEVER EVER wear them to any night spot with black lighting (trust me on this). They'd be perfect for dinner parties, or any sort of gathering at a private house with people you know well, because that's a good time to spring something a little different on everyone. "Oh, that's what he wears to parties... huh." Like that.
Key thing is to avoid looking like hotel curtains or a sofa, so (IMHO) they should be a solid, dark color, not patterned or a tartan thing or whatever (well, unless you're specifically wearing them to a Christmas party). Solid black or wine-red probably work best.
But I think what makes velvet pants *work* is conferring that unspoken ease of "he's wearing velvet pants but is just carrying on like it's an every day thing"; same rule as an Aloha shirt, really.
DH