Why not team a tan zip sweater with khaki chinos? As well as very functional, both of these items look amazing together. Dark brown suede loafers will easily elevate even the most casual of combinations. This combo is most probably what you're looking for as a middle-aged man.