It's a mistake to make inferences about elite discrimination by looking at average outcomes. Chetty recently put out an article that can inform us about this. Looking at the effect of Ivy attendance on mean income, there's nothing. But Ivy attendance greatly affects elite work:

Source: Writeup: Alternative diagram: