V neck sweaters are hideous. Ditto turtlenecks.
Don't.
V neck sweaters are beautiful and a great complement to nearly any outfit a man can wear. Turtlenecks multiply a man's wardrobe options. Both are great for layering and add warmth, both literally and figuratively. Wear them as often as possible!
There. Your option was just cancelled out by my opinion. Problem solved.
Seriously though, I think the 'v' of a V neck should only allow for the showing of a couple of buttons, or the know and the dimple of a tie if wearing one.