Confused about Allen

Why was Allen able to identify a viltrumite by face alone when confronted by Thadeus, but did not recognize Nolan when they fought in the past. It sounded like they had several encounters, and he never put it together until after his run in with Mark. Is there a reason for this or are we meant to not think about it?