So the hangup lies with us.
What gevans said above brilliantly captures the theme of this entire thread. And not just the theme, but the solution to Tim's dilemma:
The way to wear wool flannel trousers casually is to wear wool flannel trousers casually.
"The fault, dear Brutus, is not in our stars
But in ourselves, that we are reluctant to wear
wool flannel trousers on the weekend."
Many years ago, on television, there was a retrospective on President Kennedy. The program included a brief home movie of JFK at his new estate in Atoka, Virginia shortly before his assasination. In the clip, JFK is outdoors, sitting on the ground with his back against the wall of his house. He's wearing a tan crewneck sweater and what appear to be gray flannel trousers. He's feeding treats to Caroline's horse; after each bite, the horse nudges Kennedy for more. He keeps sticking his nose down into Kennedy's face. Meanwhile, JFK is ducking this way and that, and rolling on the ground, thoroughly enjoying the harassment.
When I first saw the home movie, I thought, among other things, "Why is JFK wearing dress pants when he's sitting on the grass feeding a horse?" Now, years later, I get it. It was 1963; most adults generally did not wear jeans unless they had to for work; it was a cool day, given how everyone else in the clip is dressed-so cotton khakis would not have been the best choice. JFK was never a moleskins-and-corduroy kind of guy. Wool flannel trousers? Why not? They're warm, practical, and-because they can be dressed up or dressed down-versatile.
In case you're interested, the home movie of JFK feeding the horse is on You Tube. It can easily be found by putting the words "JFK home movie Atoka" in the search field (without the apostrophes). The movie is almost 15 minutes long, but the JFK-and-the-horse part begins at 11:45.
The new estate where the scene takes place? Jackie Kennedy, who was effectively in charge of purchasing the property and designing the house, named it "Wexford." The Kennedys were able to spend only three weekends there before the President was killed. I read somewhere that JFK didn't like the place-it was too remote for his taste. It was really Jackie's project.
Other home movies of JFK are on You Tube. Like anybody's home movies, they're not that interesting. I like to watch them (with plenty of fast-forwarding) to see how JFK and his family members dressed casually in the early 1960s. There's a clip, for example, of JFK hitting golf balls (and hitting them, and hitting them...) while wearing Nantucket reds.
And it isn't just what President Kennedy wore that fits into the theme of this thread; it's what he said as well. In a televised interview in the 1950s, then-Senator Kennedy was asked how people could get involved-perhaps seriously involved-in political campaigns. He replied, "The way to get started is to get started." He suggested they begin by simply handing out leaflets door-to-door.
So get started. Put on those wool gray flannel trousers, regardless of what other people think.
Then buy an estate in the Virginia horse country.
Edit: I forgot that Tim himself said "I think it's mostly in the wearer's head." So I also want to give Tim credit for working through to a resolution. Cassius would be proud.