You can wear linen trousers with it if the texture is different. The key is to avoid wearing trousers of the same texture, so it should be something like a linen herringbone or twill jacket with plain-weave linen trousers. You could also wear a cotton/linen blend, a cotton/silk blend, tropical wool, cotton gabardine, wool gabardine (if the weather is cool enough). If the cut is more relaxed, you can pair it with chinos.