The entire quatrain where that phrase is found talks about time itself. It says that time gives beauty, thus making that particular person handsome, but time also takes away that same beauty - unfair which fairly doth excel. What was once young and beautiful will become old and wither as time goes by.
