Channel your inner James Bond and go for a waistcoat and a denim shirt. Off-duty looks for gentlemen in their 40s aren't actually that hard, as you can see here.
Denim Shirt with Waistcoat Casual Outfits After 40
Such staples as a waistcoat and a denim shirt are the perfect way to introduce some rugged sophistication into your day-to-day arsenal. A cool, less conservative ensemble for a gent in his forties.