My friend, I understand your conflicted views about wearing your flannels.I, too, lament that grey flannels are much less common and no longer really worn in casual day-to-day situations.
While I'm a reasonably confident person, I don't want my clothes to actively call attention to me, so I found the comments discouraging. Even away from the comments, I felt a bit dressed up when wearing them - I just couldn't capture a "I'm wearing chinos" nonchalance vibe. I just felt too dressed up in them to simply hand around the apartment wearing them.
I love the old pictures of college kids playing catch football in OCBDs, grey flannels and weejuns, but it's hard to capture that casual comfortableness as an outsider to what everyone else is doing. Sadly, my flannels stay on their hangers except for "nicer" occasions.
In my younger days I had the same mental hang-up about the better clothes that were hanging up in my closet. Then I asked myself, "What am I waiting for? Am I waiting for 'someday'? I'm not getting any younger; what if 'someday' never comes for me? Either I enjoy these clothes in this life, or I don't enjoy them at all. I have a choice: I can wear them, or I can keep them in pristine condition for the stranger who finds them in a thrift shop after I'm a pile of ashes. I vote for me. Upon my death, I want the clothes in my closet to be worn-out."
Time's a-wastin'. Wear the flannels and learn to love the old-world vibe. Don't overdo it, of course; keep the chinos in rotation. But now and then, make a point of padding around in the flannel trousers...just because.
(You'll find that as you age, "just because" becomes a good enough reason for doing more and more things.)