The activity or venue are paramount, as has been noted. You have said that will most likely be casual. That pretty much eliminates suits. In fact, here in sunny SoCal, getting men to wear suits, even to the opera, is a trial.
So that leaves sport coat, most likely with no tie, sweater over collared shirt, or nice long sleeved sport shirt, and complementary wool or wool blend trousers, leather shoes (no track, court, running, or gym shoes, please) in cognac/tan/russet/burgundy. Save the black leather for more dressy times, like going to the theater or a concert, when you might wear that suit, depending on where you are. Since it is getting colder in the northern hemisphere (if that is where you are), you might consider a 3/4 length outer jacket or car coat, even over the sport coat, if it's cold enough and you'll be out of doors at all. And jeans can look very smart with a sport coat, so I wouldn't nix them if the venue would call for that.
It's fall, so browns and golds and (dark) greens and rusts/brick will look like you know what you are doing. Don't go too dark if you are light complected. Harmonize with your hair and/or eye coloration (hair for outer garments, eyes for shirts). There's a plethera of that in the stores now, with anxious merchants offering hefty discounts to move the goods. To the extent possible, use what you have, since you'll feel comfortable in it and it will reflect you the person. Shift to brighter colors and more contrast in spring.
I wouldn't be too concerned about what your date will be wearing until you are going out regularly as "a couple." Do, though, be sure to let your date know what you will be doing and where going. If you avoid suggesting what your date should wear, that'll give you some idea of how that person approaches things. Of course, if you are going for a clambake and your date has never been, you'll need to coach for the basics on what to wear.
Don't overthink this, or it will stop being fun and become an onerous chore.