Help me build a small, versatile, non-business suit wardrobe

One general purpose suit that goes well with most shirts and ties is the classic grey flannel suit in an Oxford grey colour. Flannel is too warm for the SoCal weather, so a variant that might serve you well, and look good is a worsted flannel suit, or even a tropical worsted suit in this mid-grey colour. Matt S has also suggested this colour as a good option.

The advantage of this suit is that it is usually correct for most occasions, and can be dressed up with a good grenadine or foulard tie and burgundy/black shoes, or dressed down with a cream tattersall shirt and a burgundy knitted necktie or bow tie, along with country brogues or even loafers.

Beyond this, my advice is to go with blazers, sports jackets and dress slacks. This was suggested by poppies above. The advantage to taking this path is that there are many more ways in which you can combine these jackets and trousers.

Splendid plan of action!

The one additional thing I will ask that you do is to keep a single navy blue blazer, even if you decide to give others away (I'm assuming you have two or three). Ideally, this could be a hopsack or open-weave, but with the regulation brass buttons. A single-breasted model, with two or three buttons will do you rather nicely. Medium lapels, patch or jetted pockets, single or double vent -- all of these will work, and they will make the blazer wearable for a long time.

I suggest this because there may be many occasions on which a proper navy blue blazer with brass buttons will be appropriate. It is the single most essential item in any man's wardrobe. Paired with light or dark grey worsteds, or with other colours and or patterns (houndstooth, checks or plaids), it can dress you for a large variety of occasions.