Depends on what your measuring for. I don't wear undershirts so I measure without for shirts. For trousers if purchasing online beware of vanity sizing, e.g. vanity sized 36" waist may measure 37.5", slim may actually measure 36" . so I measure with shirt on. Suits, while there are variables among manufacturers take into account the jacket is to be worn over shirt.
Brooks as has been mentioned have several cuts, Madison on the boxy side, Regent , my choice, I guess you can say between Madison and Milano their trimmest cut.
But it's been mentioned there is no substitute for trying on. To answer your question and others may differ, I measure 42 bare chest and suits odd jackets in traditional or classic fit I'm 42 reg. . In Regent fit I'm 43. reg.
I require very little alteration and yes shoulder probably most important .