Which Cap?

Today it's kinda cool (upper 50's) with chance of showers, but the rain is supposedly coming to an end.
I chose to wear tan leather goods (double-monks, belt, and watch strap), brown socks, light tan chinos, a violet no-iron dress shirt, and navy blazer. Pocket square not a necessity today, but I grabbed a silk one: magenta base, with an oversize paisley pattern of a wide array of blues with a little white. Had a heck of a time deciding which flat cap to wear out in the drizzle:
Ivory cotton,
or herringbone Harris tweed in a dark blue colorway.
(Tried to get a picture of the options, but it turned into a movie of the ground and my feet... Darn cell phone. Can try again later if anyone needs the visual reference.)

I went with the blue, and did get a compliment on today's look. If it's really raining, I'll grab what's nearest / most likely to keep me dry - practicality over fashion.
But when I have the option (like today) is it generally more appropriate to coordinate headwear with outerwear / jacket? Or to grab something to complement the pants?