For a casual look with a modern finish, choose a white ripped crew-neck t-shirt and black pants. Dial up this whole getup by finishing with brown woven leather tassel loafers. This combination should clear any doubts you had about wearing casual clothes as a man in his 40s.