I'm wearing a tan (slightly greenish, actually) linen jacket right now with navy chinos, and I have to say I don't like the combination, esp. as the linen is slighty nappy, and the chinos not at all. The medium brown linen here----would look great, I'd imagine, not to mention the blue. Rust or medium gray linen blend, maybe? I like the brown linen; I think it looks suave.
For what it's worth, I'm wearing cotton glen plaid trousers from Bill's Khakis here, that register as sort of medium brown, with a tan linen/silk/wool jacket (a little more finished-looking than linen):