⇧ Having not owned a car since moving into NYC in '89, my opinions are all of the armchair theoretical type.
I've always thought it would be fun to own (as a second car) a late '50s/'60s foreign sports car. I've gone through different periods (all in my head) where I like this or that make or model more, but I always love the concept.
Having enough friends who have done so, I've come away from their experiences recognizing that it is an indulgence where the price is not tied to some intrinsic transportation value (as you note, buy a practical, thoughtfully priced car for that), but is simply a luxury good for aficionados.
You do it when all the necessary expenses are covered and you can treat yourself to something impractical.
As noted, I have no experience with them, but continue to romanticize the idea in my head.
Some favorites: