Do you have to match up the color/finish of your belt buckle with the color/finish of the bit on your loafer? Meaning, does a gold bit match up with a silver belt buckle? I tried this in the store and it just didn't seem right. This will be worn in a casual setting if this helps.