If I'm not mistaken, the origin of the lounge suit did not progress in a straight line, and the term, a suit of clothes originally did not necessarily equate to a jacket and trousers of the same fabric. When that finally did become most common, the odd jacket (Our sports coat.) and the complimentary, but non-matching trousers were born from parts of different suits worn together, often after one was orphaned.
As such, striped trousers have been worn with solid color jackets, periodically, ever since the evolution of the lounge suit, and even before that when the frock coat was the custom. And I have no problem with it, as long as the trousers and jacket compliment each other, and don't encroach too closely upon the realms of formal day dress or business wear.
There seems to be a tendency to label any stripe as a pinstripe, even when it's not. Most of the odd trousers I've seen are pencil stripes and in colors and fabrics such as those described by dfloyd, and could hardly ever be confused with the trousers from a banker's suit. I don't happen to have any striped casual trousers, but I don't know that I find anything the matter with them, the above caveats pertaining.