John Donvan
This is Open to Debate. I’m John Donvan. Hi, everybody. Across the land, a few years ago, high school students, most of them anyway, breathed a sigh of relief when colleges by the hundreds removed the exam known as the SAT from their admissions requirements. Nobody likes taking the SAT, or at least I’ve never heard of anybody enjoying it. I certainly did not. But opposition didn’t come just from the stress that students feel about it. The test has long been accused of being culturally biased and racially biased, and favoring the better off, and being unreliable as a predictor of academic success.

The pandemic comes along and provides an opportunity to make the test optional, and that’s when the trend against requiring the SAT really kicked in, and all of those schools made the change. And now, well, the mandatory SAT may be making a comeback, at least at some of the most selective schools. Dartmouth, Brown, and Yale have started requiring it again, and just before we started this recording, we got news from the Harvard Crimson that Harvard, next year, is going to reinstate the SAT, all of these schools citing the test’s actual benefits, so that we now have many schools across the nation rethinking their positions.

So, perhaps a counter-trend to the trend, and one we are going to examine in this episode’s debate, where we ask the question, should elite universities reinstate the SAT? So let’s meet our debaters. Answering yes to that question, I wanna introduce and welcome John Friedman. John is the chair of the economics department at Brown and a founding co-director of Opportunity Insights, an organization that uses data to analyze economic opportunity in this country, and last year, released a study that focuses specifically on the role of college admissions. Welcome, John, to Open to Debate.

John Friedman
Thank you so much for having me.

John Donvan
And answering no to the question, I wanna say hi to Ben Nelson. Been has made it his mission to try to rethink higher education. He founded a new university, Minerva University, which does not require the SAT or S- ACT scores as part of their admissions process. It does have an admissions test of its own. I wanna point out Ben is also on the board of Open to Debate. Ben, welcome back to the program.

Ben Nelson
Great to be here again.

John Donvan
So, before we get started, I just wanna check in with each of you, uh, uh, in a sense about your connection to the topic. And I’ll put it this way. I’ll go to you, John, first. You know, you, both of you have chosen careers, uh, that involve being around college students, and John, what, what is it that you like about teaching, about being around college students?

John Friedman
Well, what I find so thrilling is the opportunity to have young people, who are really just phenomenal and inspiring in, uh, how they got to university, uh, what their potential is going forward, uh, and from an incredibly diverse range of backgrounds. Now, I think it’s really important that we continue to expand that diversity of backgrounds that pulls students in to these schools, and I think testing is an important way to do that.

John Donvan
Okay. Thanks very much, John. Ben, what about you? What is it, uh, that you like? You s- you started your own university, and now you’re around students all the time. What’s that about?

Ben Nelson
Yeah, I mean, for me, the great thing that you see when you interact with great students, with high-potential students, is that, like every great athlete, you need a great coach, and that r- is the role of the university. And when you see students come in and then see them graduate, and see them as transformed human beings, you know you’ve done, you’ve done the right thing.

John Donvan
Ah, very interesting, and I, I wanna ask you, from the, from the department of potentially embarrassing questions, how you both did on your SATs if you remember. Ben, do you remember how you did?

Ben Nelson
I, I do remember, and it, it… And I remember it primarily because I took, I took it twice, uh, and the first time, uh, I took it, I didn’t really know what I was doing. Um, but I was of sound mind and body the second time. I think I had 104 fever. I was hallucinating and completely delirious, and I did v- vastly better. Um, and so I, I got a, I think a 790 on my, my math, but a 600 on my, on my English.

John Donvan
(laughs) How about you, John?

John Friedman
I think I got a 1530, 1520, something like that. I don’t really remember the exact details, but uh, something like that.

John Donvan
I, my, my experience, I, I actually don’t remember, except I know I was disappointed with it, but I remember what happened that day. I went to an all boys’ high school, but for the test, I was in a co-ed classroom, and I (laughs) was completely distracted by the fact that-

John Friedman
(laughs).

John Donvan
… I was in a mixed class, and it totally threw me, and I’m, I blame that for my less-than-unbelievable score. But, um, that was all a very, very long time ago. We’re in a different era now, which lets us get to our opening arguments. So, we wanna ask each of you to take a few minutes to explain the position, why you’re answering yes or no to the question, should elite universities reinstate the SAT? John, you’re up first. You’re answering yes, and here’s your chance to tell us why.

John Friedman
Well, thank you so much, and you know, just to, to summarize at the beginning, I think that, uh, these universities should require standardized test scores because, uh, they are the fairest and broadest means to let students, no matter their background or, uh, life experiences, demonstrate that they have the academic preparation, uh, and potential that these universities are looking for when they, uh, do an admissions process.

Now, that argument, I think, relies on, uh, three different parts, and so, uh, I’ll just lay them out. The first one is that standardized test scores are predictive of students’, uh, academic success in college, their success beyond college, in ways that, uh, hold even when you just compare students with the same race, gender, and, uh, socioeconomic class. Uh, so this is really picking out something that’s, uh, really important for schools that care about academic preparation, and it’s doing so, especially at these elite schools, in a way that, uh, even high school GPA doesn’t. Uh, there’s really, uh, no other broad-based means that has the same type of, uh, predictive ability.

Now, that predictive ability wouldn’t, I think, be much help if it were biased against students from different backgrounds, and so that comes to the second point. In recent work that I and others have done, uh, we’ve directly examined this in the data, and what we find is that these tests don’t seem to be biased against students from, uh, lower income or less-advantaged backgrounds. Now, what do I mean by that? Specifically, uh, take two students that just get the same thing on the test. Let’s say they both get a 1400. One of them comes from a really well-resourced high school. The other one doesn’t.

Now, if the student from the less well-resourced high school, with that same score, ends up doing way better than the student from the, the more well-resourced high school, you know, maybe because that, uh, student had the opportunity to study a lot more or take the test more times, we’d say the test is biased and that it’s not really fully capturing the potential of the students from the less well-resourced high school. But, when we look at the data, we see that those two students, uh, on average perform really similarly when they get to school. Uh, they have similar outcomes after college, and this isn’t just true depending on the high school you come from. It’s true if you look across students from different socioeconomic classes, from students who are first-gen or not. The test really seems to be providing, uh, not a perfect, for sure, but a, a clean estimate of the academic preparation, that’s a really important input into these admissions processes.

The third aspect of this, and this has become especially important in recent years, is what happens when schools give students the option to submit their test score. And I get the intuition you wanna let students choose, uh, what pieces of information to put forward, but in practice, it turns out to be another level of strategery, another layer of complexity in the admissions process, as students are trying to figure out, “Should I submit the test score or should I not submit the test score, and is it gonna help my case?” And too often, students who are precisely those we’re trying to attract, students from, uh, backgrounds where they don’t have lots of friends and family who have gone to these schools. They might not have a college counselor who’s really experienced in, uh, a- advising students on how to apply to these schools. Those students will, uh, see some very high average tests scores and conclude that their test scores won’t help their application, even when in practice, they will.

So, I think when these schools return to not just allowing the option of test, uh, submission, but requiring it, it’s actually leveling the playing field in a way that creates the fairest way to assess academic preparation, that in combination with many other factors, will determine admissions. So I think that’s the argument for requiring testing at these elite schools.

John Donvan
Thanks very much, John, and now, Ben Nelson, it’s your turn. Your answering no to the question, should elite universities reinstate the SAT? Please tell us why.

Ben Nelson
Sure, happy to. I think first and foremost, we have to understand what are the roles that elite universities fulfill in society. And by and large, these institutions are signals. They’re filters. And the reason for that, and we know this, uh, and we have significant evidence for it, is that elite universities do not actually provide a better educational product than non-elite universities. The reality is is that physics is taught the same at Harvard as it’s taught at Beacon Hill Community College down the street. And in fact, there’s ample evidence that professors who do not focus on research do a better job at teaching than those that do.

And so the real value add is a selection and a signaling. And now we have to ask, what is it that universities select for? And the reality is is that universities, in their entire application process, select for wealth. Now, it is true that some components of the application process select for wealth even more so than others, so for example, the pre-written essay that universities love to rely on is more correlated with wealth than SAT scores. But it cannot ignore the data behind SAT scores and how unbelievably skewed they are.

So, we heard data where you compare like for like scores and what happens in college, and there’s a problem in both sides of that. First and foremost, what is the opportunity for like for like scores? Students born into the top 20% of the socioeconomic distribution have an 80% propensity at taking SATs. And of those that take it, 17% of them will wind up getting scores of 1300 or above. Those that are born into the bottom 20th percentile, the bottom quintile of the socioeconomic distribution, only take the SAT at a rate of 20%. And of those, only 2.5% get scores of 1300 or above.

Now, we can say, “Well, those that already do well wind up do- having a decent outcome after all,” but the issue is those that are excluded from the beginning, both because of cost, both because of various biases in the test, and because when you actually look at the dollars invested in test preparation that helps with, uh, increased scores, et cetera, people from lower socioeconomic backgrounds simply do not have the access that is requisite. Now, the reality is that there are better approaches, but it would require universities to actually be committed to true socioeconomic diversity in their student bodies.

And just to give the perspective on what the outcomes are and what we did at Minerva University in comparison. If you look at an incoming Ivy League class, only about 13% of the incoming student body comes from households making less than $50,000 a year. At Minerva, where we don’t have SAT optional, we refuse to look at SAT scores even if they’re submitted, and we have our own assessments, 60% of our student body comes from households making less than $50,000 a year. Same standardized approach, but one that doesn’t lend itself to wealthier students doing better.

John Donvan
Thank you very much, Ben Nelson, and now we have heard from both of you, and we know where you stand and why, so we’re gonna take a quick break, and when we come back, we will dig deeper into the question, should elite universities reinstate the SAT? I’m John Donvan, and this is Open to Debate. Welcome back to Open to Debate. I’m your moderator, John Donvan, and we’re taking on this question, should elite universities reinstate the SAT? We’ve heard opening statements from our two debaters, Economics Professor John Friedman and Founder of Minerva University Ben Nelson.

I just wanna go over what I heard you say in your arguments. John, you’re basing, uh, your argument very much on your own research, and the research has been widely published and been quite influential very, very recently, but you’re making the case that against, uh, accusations that the SAT produces biased results, that that’s not actually the case. You point out that while it’s not a perfect measure, and there is no perfect measure, it provides a pretty clean estimate, that was your term, for the potential of students to do well at university, and also after, in life. And you suggest making the test mandatory, not optional, because the decision-making process to submit or not submit works to the detriment of students who might otherwise do better if they submitted lower scores.

Ben Nelson made, uh, the argument that the whole process, uh, of, uh, college admissions s- as he puts it, selects for wealth. In terms of the test itself, his, uh, fundamental argument is that the test kind of plays to the strengths of people who are better off, because they can pay for test prep, et cetera. So, I wanna go b- first to you, John. Ben’s overall framing that college admissions is selecting for wealth, what do you make of that?

John Friedman
So, I agree with that. I think that there are many aspects of the admissions process, whether that be the essay that Ben mentioned, the preference for students at these schools who are gonna play on athletics teams, who my research, uh, showed disproportionately come from high-income families, the preference given to students who are legacies, the children of, uh, previous alumni, um, and even the reliance on general things like personality and extracurriculars. All of these things are aspects of applications that serve to increase the rate of a- admissions for students, uh, from, uh, high-income families, and lead to exactly the type of, uh, skewed class that, uh, Ben mentioned.

The one thing that actually seems least biased in that entire process is standardized test scores. Now, Ben mentioned that there are these enormous test disparities between students from high- and low-income families, and of course, that’s exactly right, but I don’t think that results from biases in the test itself. That’s a statement about deep underlying inequalities in the c- uh, schools that these students attend, in the neighborhoods and other environments where they grow up. If you look at the test score disparities at other tests that students have taken along the way, like the NAEP scores, where students don’t take it twice, they don’t study for it, they don’t pay for tutors, the disparities are almost exactly the same as those we see in standardized test scores. And so again, it’s not a perfect measure, but at the moment, it’s the fairest and broadest measure that we have.

John Donvan
So, so John, you’re, are you saying that of all of the kinds of ways that admissions departments can decide to assess a student, this is the one that is most likely to rescue a student who’s disadvantaged from being just lost in the shuffle, being able to turn in a decent score, especially a decent score relative to the community that they come from?

John Friedman
Uh, yeah.

John Donvan
Is that-

John Friedman
That, that’s right.

John Friedman
Specifically on the dimension of academic preparation. Obviously, a test score, uh, per se, doesn’t tell you anything about whether the student is curious, whether the student, um, is, uh, someone who brings other qualities that, uh, academically or otherwise, the schools are looking for.

John Donvan
So let me bring that back to you, Ben. Uh, John is, is not arguing that it’s perfect, but that despite that, uh, among everything else that’s out there, it’s the closest that comes to a sort of uniform, potentially objective measurement of what a student can do.

Ben Nelson
Well, uh, uh, and again, I don’t disagree with the fact that it’s better than what the admissions departments are actually using today. The difference is, is that what admissions departments should be doing is actually rethinking the admissions process overall. I don’t disagree with the idea of standardization. You know, it used to be, 30, 40 years ago, the early tests of the SAT and if its, and effic- and its efficacy, uh, work by Jonathan Barron and others in the 1980s m- um, moving forward, showed very little, uh, predictability in SAT scores, and it showed quite high predictability when you look at GPA and what at the time was called achievement tests-

John Donvan
When you’re-

Ben Nelson
… which were [inaudible

]

John Donvan
And, and you’re talk- Just so people get com- comfortable with the term predictability, you mean does the test predict how well a student’s gonna do as a-

Ben Nelson
In college.

John Donvan
In, in college, yeah. Yep.

Ben Nelson
Exactly.

John Donvan
Mm-hmm.

Ben Nelson
And, and-

John Donvan
So you’re saying that the-

Ben Nelson
The, well-

John Donvan
… that that study suggested the SAT was not good at it, and the GPA was much better at it.

Ben Nelson
The, at the time. GPA in combination with achievement scores, and achievement scores were standardized tests, and so standardized tests do have a, a role. Uh, having said that, there were two fundamentally different conditions at, at that time. Number one, GPAs gave you some real data about high schools. They don’t anymore, because of massive grade inflation. But that’s also true of college grades. Uh, my, my favorite little e- uh example used to be that in 1963, Yale University only awarded 14% of its grades as As, and by 2013, it was 68%. Now, the, um, New York Times and Wall Street Journal reported that last year, it was 79%. And so-

John Donvan
Well, that’s just students-

Ben Nelson
… you can’t-

John Donvan
… getting much, much smarter than they used to be, right?

Ben Nelson
Oh, oh absolutely. Even with that, however, you do see some real problems with this test. So for example, gender biases. And this is data from the ETS itself. Female test-takers of the SAT get 33% fewer points on the math section than male students do, that then have the same grade in their college level class. Female students who have better grades in math classes overall in high school get lower scores on the math SAT. Not to mention the fact that accommodations, so-called, uh, things that where you get 50% more time to take the test, is vastly more prevalent among wealthy, uh, applicants than non-wealthy applicants, once again.

John Donvan
So, let-

Ben Nelson
Sometimes of course-

John Donvan
… let me-

Ben Nelson
… students-

John Donvan
… let me-

Ben Nelson
… deserve accommodations but-

John Donvan
… let me take a little bit of-

Ben Nelson
… oftentimes, they do it falsely.

John Donvan
… of, o- o- of the, of the point I think you’re making to John, which is that while in theory, the test may present a relatively level playing field, that in practice, uh, that’s, that’s kind of not what’s happening. I’d like to get back to the gender gap a little bit later, but that in practice, because of the ability of people to, uh, to do test prep and, you know, to take the test multiple times because they can pay the fees, that in practice, the SAT is, uh, is not really a- as, as flat as you’re saying. And his second point is that if your metric is success (laughs) in school, he’s just arguing, well, the, the database of GPAs in the modern era is so diluted that it’s really hard to make that claim. So I’d like to ask you to take both of those points on.

John Friedman
There’s been a lot of research about the relationship of these, uh, test scores and high school grades to, uh, outcomes in college. Most of that work includes a very, very broad swath of students attending a very wide range of schools, from elite universities, to, uh, selective universities, to essentially open access four-year universities. And, uh, it’s pretty clear that you see in the data that the relationship between, uh, test scores and high school GPA and, uh, students’ outcomes in college, they are actually not the same across these different settings. And what you see very clearly is that at these elite schools, high school GPA is, uh, really not predictive of outcomes in college, while, uh, test scores are, and that’s even in the presence of the, uh, recent grade inflation. If you look at grades that are a C or below, uh, now that doesn’t happen very often at these schools, but, uh, in their first year, students who have, uh, lower test scores are much more likely to academically struggle by this measure.

Now, the second point, about all of the different resources that students put into the test, uh, that’s right. Students really do try to, uh, do everything they can to, to game this, and it’s not that you can’t game it, but again, in the context of students who are scoring, uh, towards the top of these ranges, the data suggests that that stuff doesn’t really have much of an effect. Uh, there’s the statistic that I quoted, where two students from very different backgrounds, with the same test score, do not on average perform differently, right? If you’d expected, uh, the, the gaming to really matter, then the student from a high-income family with a 1400 would actually perform considerably worse, uh, once they get to college than a student from a low-income family with a 1400, because that student from a high-income family had artificially inflated their test score. But again, that’s not what we see in the data.

The second piece of evidence is that there’s, uh, good work studying what happens, uh, how much of a test score boost you get when students take the test a second time. And while they find quite large increases, of 100 points or more, for students that start out, uh, from relatively low scores, like 7 or 800, they find that the increases are much smaller, uh, and in some cases not even statistically significant, when students are starting up in that 1300, 1400 range, uh, where it’s really, uh, students start to, uh, and the admissions rates, uh, start to be quite a bit higher. And so-

John Donvan
All right, let me, let me… Uh, I wanna bring it to Ben, but um, however you’d like to respond to what he’s just said.

Ben Nelson
Yeah, absolutely. So, so a few things to understand. Dynamic range really matters at the top. So, it is true that obviously, when you start at a much lower SAT score, you’ve got a lot more headroom where test prep and multiple tests will, will help, and you know, many test prep, uh, organizations will say, “We will raise your score by 200 points or your money back.” Of course, it’s extraordinarily rare for somebody to be able to go from a 1410 to a 1600. That’s, that’s, uh, almost unheard of. However, small increments at that range matter quite a bit for college admissions, right? And so the difference between, for example, the, a 1410 and a 1440, when you apply, especially depending on distribution, right? This is particularly true because when universities report out for these rankings, which unfortunately drive so much of higher education, they look at the 25th to 75th percentile of the SAT range of the incoming class, and that has, that drives a lot of behavior. And so, even small increments, which absolutely occur when you take tests multiple times, when you do test prep, really do matter. So, so that, that is one factor.

And the second is, it, and, and I’m actually curious for, for John to, to expand more on it. On those high test scores and their performance in college, there, I would imagine, the dynamic range in the college, uh, uh, GPA matters, right? So if everybody is in the dean’s list or, you know, magna cum laude, then sure, you’re not gonna see much of a difference between the w- wealthy high test score and the non-wealthy high test score, so I- I’m just curious as t- as to, as to w- how much dynamic range there is.

John Friedman
The difference between the rate at which students with a 1600 SAT score versus a, you know, 1300 or a 1200 SAT score, have this type of academic struggle is, is pretty big. Uh, ranges from about, uh, essentially no one, maybe 5% of students with a perfect score, uh, you know, up to, uh, you know, 30, 40% of students who have stud- uh, scores in that 1200 to 1300 range. When you look at the difference for students with a given score, so let’s now just look at students with a 1400, uh, the difference between the rates of academic struggle for students from high- versus low-income families, or from students from first-gen versus non-first-gen families are very small, on the order of a couple percentage points. And, and, and actually what you see is that the, the, the students from high-income families are, uh, struggling a little bit less. I totally agree with you that schools should be pulling in more students from a much, much broader range of backgrounds, but because of these broad inequalities in society, doing so requires a trade-off between that type of academic preparation and, uh, you know, the diversity of background, and I [inaudible

]-

John Donvan
John, we, we, we, we kn- we know that schools do wanna have a diverse background, but we also know that they’re not allowed to use Affirmative Action anymore. Where does the, uh, having the SAT in the mix play into that dynamic?

John Friedman
The Affirmative Action, uh, ban applies only to writs, and so for a school that wants to, let’s say, expand the number of students coming from, uh, low income, or students who are PELL eligible, uh, this decision is, uh, totally irrelevant, um, and students, uh, the schools are free to admit as many, uh, low-income students at whatever level of academic preparation they would like, and, um, I, I would encourage them to, to admit more. Uh, there’ll be less racial and ethnic diversity than before, but we still face the same trade-off. Uh, schools are trying to identify students from these underrepresented backgrounds who they really feel can thrive and succeed at these schools, and, uh, you know, I think the, uh, having a strong test score is one of the fairest ways that those students can signal that level of preparation.

John Donvan
Ben, um, you’ve made it clear that you don’t like the SAT, but you do believe in standardized tests. So, at Minerva university, which you founded, what, what is your, what is your approach that’s not the SAT, and what’s wrong with the SAT that your approach is addressing?

Ben Nelson
So, there’s two core aspects about the way in which we do our assessments at Minerva University which differ from the SAT. We have a battery of challenges that we give our students. It’s a bunch of questions. Some are automatically graded. Some are manually graded. Those that score them don’t know the rubrics. They don’t know the weightings. There’s a bunch of things, red herrings, that we throw in there. And everything is done secretly in the background.

John Donvan
So, one advantage you’re saying over the SAT is that it’s much less gameable. It’s, you’ve created a-

Ben Nelson
Correct.

John Donvan
… a, a way that cannot be gamed or…

Ben Nelson
It’s much harder-

John Donvan
… trained for.

Ben Nelson
… to game.

John Donvan
Okay.

Ben Nelson
And, because we changed it year to year, when you think you know how to game it (laughs) now the following year, you cannot game it. And, and I think that, that this is important for elite universities. If elite universities were truly committed to socioeconomic diversity, they would get rid of the pre-written essay. They would g- be really overhauling things.

John Donvan
Okay, let me… That’s a very interesting proposition. I wanna take it back to John. Ben is suggesting that perhaps Brown could go a better way by not doing the SAT, but doing something standardized and designing it for itself. What do you think of that?

John Friedman
So, if we have our own bespoke thing, and you have your own bespoke thing, and Harvard has its own bespoke thing, I really start to worry about whether students who, you know, don’t live in this world, uh, they might not even be thinking about attending these types of schools until they’re really performing strongly, um, towards the end of high school. That’s where the access barriers really, uh, crop up, and I, uh, th- that holds, uh, just as well for these type of bespoke tests as for the type of more advanced tests, like the, the subject tests that we don’t have anymore. I think the example, uh, in, in this day and age is the AP tests. AP tests are great predictors, and I’m happy to, uh, have that be part of the process, but the set of students that have access to a broad range of AP tests and AP classes to, to, to learn the material to prepare for them, is much, much narrower than the set of students who, uh, have access to, uh, standardized test scores like the SAT and ACT.

John Donvan
And in addit- in addition, Ben, uh, you know, nowadays students, because the common app, et cetera, they’ll, they might apply to 50 colleges, so-

Ben Nelson
Exactly.

John Donvan
So-

Ben Nelson
And, and that’s why universities refuse to do it.

John Donvan
But-

Ben Nelson
Look, we-

John Donvan
Uh, uh, but, but, wait one moment. If they wanted to apply just to three or four, and each one had its own separate exam, as opposed to taking one test, maybe one test more than once, but one test, they’re gonna have to take four tests, very, very big, lengthy process, and that would, I think would curtail their opportunities to apply to more colleges than one, and especially people of more challenged means.

Ben Nelson
So, we present evidence, real evidence that that’s not the case. Minerva is a brand new university. We’ve only graduated five classes. And as of three years ago, we have 25,000 applicants. Now, granted, that may not be the 40 or 50,000 that the Ivy League universities get, but it’s not that far (laughs). It’s also important to note, our applications are free. So, so it is an investment that we make in order to open up that catchment area, but that is exactly why universities don’t like this, because universities love the common app. They love the fact that you, that students apply to 20, 30 different universities automatically, so that they can reject many more students and appear more selective.

John Friedman
Yeah, I just wanted to jump in, because I think, uh, and I totally agree that schools care about their admissions rates, but moving back to requiring standardized testing is gonna reduce the application volume by 20 or 25% in all these schools. We saw that when MIT went back to requiring testing two years ago, um, and so I bet that if you were to replace your bespoke admissions test with one that was just the SAT or ACT, it’s great that you’re getting 25,000 applications now. I bet you’d get up to 35 or 40,000. And obviously, that’s your, your choice to do that.

Ben Nelson
Uh, we, we very well may.

John Friedman
You know, I think, uh, you know, that it’s just not plausible to argue that bespoke admissions test create the same type of open access to students from all backgrounds as, uh, as these standardized tests.

Ben Nelson
Well, uh, but again, the, the, the, the proof is in the pudding, right? Five times more socioeconomically diverse student body at Minerva than at any Ivy League university.

John Donvan
We’re, we’re coming up to a break, and there’s one topic I want us to g- to get back to, and that was, John, the gender gap that Ben mentioned, that the r- the results from the Educational Testing Service itself have shown, at least in the past, you might have updated information on this, that female students, uh, girls do m- more poorly o- overall on the SAT, especially in math, than boys on the SAT, but they’re doing better in the classroom, and they’re even doing better in college, and that, in that sense, there’s, the SAT predicts lower results for girls than they actually get in the real world, and that that’s an established gender gap, and I’d like you to take that on.

John Friedman
Yeah, so we don’t find that type of, uh, uh, disparity in predictions when we look in, in the, uh, at our data for students performing at standardized, uh, at these elite schools. Now, one thing that I think you may need to factor in is that these schools get, uh, way more applications from women than men. Um, and then they actually have, uh, it’s actually Affirmative Action for men in order to get the admissions, uh, admitted class back to something that’s closer to 50-50.

John Donvan
All right, we’re gonna take a break now (laughs) and when we come back, we’re gonna continue our conversation around the question, should universities reinstate the SAT? And when we come back, we’re gonna have some more voices join the conversation. We’ll be right back. I’m John Donvan. This is Open to Debate. Welcome back to Open to Debate. We’re taking on the question, should elite universities reinstate the SAT? I’m John Donvan, and I am joined by our debaters, Economics Professor John Friedman and Founder of Minerva University Ben Nelson. And now we wanna bring in some other voices, people who’ve followed this issue, who study this issue, who write about this issue, and first up, I wanna welcome Janet Lorin. Janet extensively covers h- higher ed at Bloomberg News. She’s been doing that since 2008 there, also won the George Polk Award for National Reporting for a groundbreaking story about student debt in 2012, and as we went to recording today was one of the first reporters to break the news that Harvard is going back to the SAT, and will be doing so in 2025. Welcome, Janet, to Open to Debate, and, uh, we’d love to have you come in with your question.

Janet Lorin
Thank you, John. Uh, so there’s been certainly a cascade of elite schools going back to the SAT, as you mentioned Harvard today. We talked about gamesmanship in schools trying to get lots of applications to look more selective. I’ve written about that extensively, and that is true. However, I’ve written about another topic including recruiting rural students. A host of colleges have done this this year, including Brown, MIT, and Yale, and I understand what you were saying, that you need to see a student in context. An 1100 in a rural area may be quite high if compared to their peers. Uh, but I go back to something that Yale’s dean of admission told me that is sticking in my head, which is students have to show that they can do the work. They, when they get to a place like Yale, or Harvard, or where- wherever else, that they can do this high-level work in college classrooms. Do you think doing well on these test scores will help students succeed when they get to these very competitive places?

John Donvan
John, I think the question’s directed-

John Friedman
Sure.

John Donvan
… initially to you.

John Friedman
So, uh, I don’t think that the mere act of doing well on these test scores has anything to do with succeeding academically once you get to college. It’s that the tests are measuring, imperfectly, but they are measuring an underlying level of academic preparedness, that that is what helps the students to succeed at these schools. It’s not that the test is a perfect way of doing that. It’s just that when you look to other ways, how would we take a student from a rural high school and really be sure that they can do the work? High school grades aren’t nearly as good. High school grades are four times less predictive of academic success in college or academic struggles in college as standardized test scores. Access to AP tests means that those too are not as broad-based a measure, uh, as standardized test scores. The SAT, uh, is being redesigned this year. We’ll hopefully have more data going forward. Maybe that is a yet-less-gameable version. That’s certainly what they’re going for. I don’t, I don’t know at the moment, um, but in terms of the tools that we have available, uh, right now, for admissions offices to use, I think, uh, the, the SAT and ACT tool- scores are the, the broadest and fairest way to, uh, measure that underlying level of academic preparation.

Ben Nelson
Maybe they, they, they add a little bit of additional color on, on being ready, uh, for, for work. I, I once gave a, a talk at Harvard a few, uh, years ago, where I was making an argument that, you know, u- universities need to get rid of introductory level courses, you know, professors reading a textbook out loud. There’s no point. And, uh, one of the deans at Harvard, um, came up to me afterwards, and he says, “Ben, you, you don’t really understand. Um, every year, we have students, admitted several students, admitted to Harvard, matriculated to Harvard, that cannot give the correct answer to the following question, which is 2X = 16. What is X?”

Now, I was… Uh, I didn’t believe him. I thought he was kidding, and he says, “No, no, no. This is not an example of a question. This is a question we ask, cannot give a correct answer to it.” And to, uh, John’s point, those that are failing, uh, or getting a C, ’cause you can’t fail at, at Harvard. It’s an impossibility. You can’t get an F in a class. But, uh, for those that get a C, the gentleman’s C, um, in, in these institutions, that, that is largely, uh, uh, indicative of that. However, you, you might wonder, well, how do these students show up at these institutions? You know, how d- does somebody with an 1100 SAT score show up at these institutions?

And, and the reason is is that daddy built a building, whereas there may be others, and for your example in the rural setting, that may be getting a lower SAT score and would be doing just fine, right? So even in, in John’s work, if you’ve got, you know, a 30 to 40% failure rate for, or failure, C rate, uh, under-performance rate for those students that have low, low SAT scores, you have 60 to 70% that are doing fine, right? And so, and so that really is the question, which is given that level of difference, is it actually the SAT that is the predictor, or is it a bunch of other factors that is obscured by the admissions office?

John Donvan
Thanks, Ben, and thank you very much, Janet, for your question. I now wanna bring in Liam Knox. Liam covers admissions and enrollment at Insider Higher Ed, and also writes their weekly newsletter, Admissions Insider, which you can find on the, uh, Inside Higher Ed website. Welcome, Liam. Thanks for joining us, and, uh, what’s your question for our debaters?

Liam Knox
Thanks for having me. It seems, I think, to the average person that until this year or last year, the overwhelming consensus was that standardized testing privileged students with more resources and more money, often white students, but in the past year, uh, research cited by elite universities as they make their testing policy decisions, including, John, your research and the research by your colleagues at Opportunity Insights, uh, has come to different conclusions on its impact on diversity, and then some universities have come to completely disparate conclusions on its impact on diversity. Um, Harvard and Dartmouth cited research that found it more often helped underrepresented minorities stand out in the applicant pool. The University of Michigan’s research team found that the opposite was true. Now, Michigan, long considered a, a public Ivy is not exactly an open access institution. It’s, it’s very elite itself in its own way, and it’s not the only elite institution to go test optional and, and to support it with research. Uh, Columbia, William & Mary, there are a few others. [inaudible

].

John Donvan
So it sounds like it’s all over the map.

Liam Knox
It’s all over (laughs). It is definitely all over the map. Frankly, it’s, uh, uh, the state of play is, is anything but standardized, I would say.

John Donvan
Okay, so, so having laid that out. What is the question?

Liam Knox
The question I have is, uh, where is the divergence coming from in this research, that elite schools are doing internally and coming to totally different conclusions? Do we really have enough relevant data to be confident one way or another?

John Donvan
(laughs) All right, thanks. John, you’re the economist. Um (laughs) please-

John Friedman
Yeah.

John Donvan
… tell us why the numbers don’t s- see, tell the same story.

John Friedman
Well, I think, uh, what we’re seeing is the importance of context. So, let me just give an example. At, uh, schools like Brown, we have students applying from all over the country and all over the world, in a way that, uh, high school grading systems may be extremely different. I think that’s part of what contributes to the lack of predictive power if we’re trying to assess academic preparation with something like high school GPA. If University of Michigan is now comparing only students that are at different high schools, uh, probably public high schools within the state of Michigan, uh, that now may be a much more standardized, uh, set of grading policies that, um, make the value of, uh, high school GPA quite a bit different.

Um, these schools are also, uh, admitting students at different levels of selectivity, uh, with different average test scores, right? You, you can go online and you can see how the, uh, the test score range for a place like Harvard is different than for a place like Michigan, is different from a place like, um, Central Michigan, and I think especially because these institutions do have the ability to look in their own data, to judge these different things, it’s really important to be data driven in, uh, how one sets policy, and, uh, you know, I’ve seen enough data from enough different universities that I think what I show in recent work is pretty common across, uh, Ivy+ schools. I would expect it, uh, and I’ve talked to people, it also seems quite similar in other, uh, elite private universities. Uh, but things do look a little bit different when you look into the, the public Ivies, the, you know, the very top public flagship schools in the country. Uh, it’s okay if, if the policy differs, if it’s data driven and appropriate for the situation.

Ben Nelson
And maybe I’ll, I’ll add a little bit more, uh, on that. I mean, University of Michigan is actually a great example, because the, such a large number of their students come from outside of Michigan as well. I mean, the University of Michigan, I think some, uh, was some years, 50% of their undergraduate student body, uh, came from out, out of the state. And, and it is a context-dependent question. I think to John’s point, compared to the pre-written essay, the SAT is less socioeconomically biased (laughs). It doesn’t negate the fact that the socioeconomic bias (laughs) in the SAT has been shown for decades, uh, to be significant, right? And so you can cut the data a lot of ways, and whenever you use correlative rather than causal data, you- you’ll be able to get a lot of, um, uh, disparate types of interpretations.

And, and that is actually the crux of the issue, which is that if you are stuck in the assumption that the rest of the application is correct, right? That you should be admitting legacy students (laughs) and, you know, g- varsity athletes, which make up, at places like Harvard, 20% of the incoming class, places at like Williams, 50% of the incoming class, right? 200 high schools, uh, in the country, and you can guess what zip codes those countries are from, represent 50% of the incoming class of the Ivy League, right? So if you assume that all stays the same, sure, any standardized (laughs) uh, uh, test is gonna give you less bias (laughs) than, than what you have.

John Donvan
Liam, thank you very much for joining us, and I next wanna bring in Derek Newton. Derek’s a freelance journalist on education and also a contributor to Forbes. He has his own Substack, The Cheat Sheet, which explores issues of education. Derek, Cheat Sheet is the best Substack name I’ve heard so far, uh, so please come on in with your question.

Derek Newton
Well, thank you. Uh, thanks for having me, uh, I guess my question i- is somewhat rhetorical, but also, uh, I think important. If we know things about all these, all the data that we collect in the admissions process, you know, we’re learning the correlation to grade point average in high school, we understand that the SAT score may mean one thing if you’re from a certain background or a rural setting, we understand the bias that’s baked into an essay. I still come at this that the admissions decision is im- is heavily human, and very complex, and so my question is in what setting where people have to make a complex decision do we benefit by having less information?

John Donvan
Just so I understand, Derek, who are you pointing that question to more? Because John is not really arguing for less information.

Derek Newton
No, no. Right. I, I mean it, it, like… As I said, it’s rhetorical. I guess Ben, you know, uh, he has his, his institution has their own test, but to the question of whether the schools should have the SAT, the SAT exists, students take it. Why would you not want that information in making a complex decision?

John Donvan
Yeah. Ben, let’s take the question to you, ’cause it sounds like why, why not?

Ben Nelson
So, the, the generic answer to why less information is if the information is misleading, you want less of it (laughs) right? And so that’s one category. Second category is how you use that information, right? There could be some threshold, um, value, and there’s, there’s without question threshold value in the SAT, right? It is likely something where if you’re scoring, you know, below 1000, you really shouldn’t be admitted into, uh, an, an elite university, right? That, that’s an easy, and maybe the, the cutoff is 1100 or, or what, what have you, right? So, so there is broad usage of, of information that can be gleaned from it.

The problem is when you misuse information that is biased, or that the information is misleading, right? So for example, if you’re making decisions that say, “Hmm, you know, this group of people is getting 1390 and this group of people is getting 1410, and optically, 1390 is not nearly as good for us as 1410, and therefore, we’re going,” uh, you know? And then, when you look at the, uh, again, the dynamic range of those individuals that’s can s- that have taken the test, 1390 in their first shot versus 1410 in their fourth shot, right? Uh, then you’ve got, you’ve got information that is, that is, uh, deleterious to use.

John Donvan
John, uh, you’re gonna get the last word in this round.

John Friedman
Uh, I mean, I, uh, that doesn’t sound like, that, that kind of, you know, “Let’s admit all the people with a 1390 and let’s,” uh, or sorry, “Let’s admit all the people with a 1410 and (laughs) and reject all the people with a th- with a 1390.” That doesn’t sound like any, uh, admissions decision, uh, that, that I’ve, uh, sat in on, and you know, I’ve tr- tried to observe this quite a bit. Uh, my sense is that, uh again, just speaking about these elite schools, they’re, uh, using this information in context. They’re using it as part of a suite of other things, so that, uh, you know, it’s not that a rural student who gets a 1200 is gonna outperform a student from, uh, a wealthy part of a city with a, a, with a 1400. It’s that that rural student is bringing, uh, uh, lots of other, uh, value to campus, even if they, uh, will on average perform, uh, less well, or, or, or struggle more. And I think this, uh, th- these test scores, uh, help, uh, the schools make this type of trade-off as, as best they can, by and large.

John Donvan
Derek, thank you very much for your question, and that brings our discussion round to an end, and now it is time to bring this all home with our closing round, which are closing remarks. John, you are up first, uh, to make your closing statement. One more time, explain to us why you are arguing that elite universities should reinstate the SATs.

John Friedman
Well, thanks John and thanks Ben. It’s been a great discussion. Sometimes, uh, you know, with all the statistics that we both cited, I think we can lose the, the human side of this, and so, you know, I just wanna, uh, talk about a, a young woman named Anna. Uh, she’s a PELL-eligible student. Uh, she’d be the first person in her family to attend college, uh, and she grew up in a pretty high-poverty neighborhood, commuting an hour to school each way in order to, you know, try to just find the best opportunities for her, despite the fact that she’s working 20 hours a week to support her family.

Now, despite the fact that she went to the best school that, you know, she could, uh, get access to, still, uh, you know, this is not a, a well-resourced school, and just getting really good grades at that school, uh, might not be enough to, uh, really give the admissions office the confidence that, uh, that you, you know, she’s gonna be able to, to do the work. And so I think what was really helpful in this case is that Anna chose to submit her test score. It was 1430. Um, and, uh, it’s strong enough to demonstrate in her context that she is, uh, a student with amazing potential.

Now, without that test score, I’m not sure if the admissions office would’ve had that confidence to admit her, and because of that, I’m really thankful that Anna chose to submit the test score, unlike so many others who, especially from that type of background, they might get scared that it’s not a perfect score and it’s gonna hurt their chances, when you know, in this case, you know, Anna having a, a 1430 on her, uh, admissions application was incredibly helpful. And so it’s, uh, it’s really because of, uh, trying to make sure we can identify and recruit students like Anna that I think, uh, it’s, it’s great news that we’ve seen, um, uh, a number of, uh, these elite schools return to requiring, uh, test scores, uh, in recent months, um, and I hope that the trend continues so that we can, uh, make sure that we find lots more students like Anna going forward.

John Donvan
Thank you, John, and now Ben, your turn. You’re here to tell us one more time why elite colleges and universities should not reinstate the SAT.

Ben Nelson
So, again, thank you, both John and John, uh, for a great conversation. And, and I think that it is crucial to remember, uh, just as we started, the role of these institutions. It simply cannot be that you have a six times more higher propensity to demonstrate your intellectual capacity if you’re born to the wealthiest members of society than if you were born to the least wealthy members of society. And that is what the SAT allows for, a six times rate of success, 1300 or above, with a propensity to take the test that is four times greater. And so, the reality is is that this standardized assessment floods the zone with individuals coming from privileged backgrounds, that is not correlated to the IQ distribution in society, right? If you have that level of applicants, 24 times more likely to demonstrate between a wealthy applicant and a poor applicant, then something is clearly wrong with that measure. Now, it may be that other measures are even more flawed, but these universities are extraordinarily resourced. They have been trusted by society to select individuals that have the highest potential, and they owe it to come up with a better solution than what the SAT can provide.

John Donvan
Thank you, Ben, and that is a wrap on this debate. And I wanna thank both our debaters, John Friedman and Ben Nelson. We so appreciate your showing up for this, uh, approaching the conversation with an open mind, actually listening to one another, uh, bringing thoughtful disagreement to our table, in short for being Open to Debate. I wanna thank you both for being here.

John Friedman
Thank you so much.

Ben Nelson
Thank you.

John Donvan
And I also wanna thank my fellow journalists and experts for bringing their provocative questions to the table, so thank you, Janet, and Liam, and Derek. And finally, a big thank you to all of you who are listening in our audience for tuning into this episode of Open to Debate. You know, Open to Debate is a nonprofit that is working to combat extreme polarization through civil debate, and our work is made possible by listeners like you, by the Rosenkranz Foundation, and by supporters of Open to Debate.

Robert Rosenkranz is our chairman. Our CEO is Clea Connor. Lia Matthow is our chief content officer. This episode was produced by Alexis Pancrazi and Marlette Sandoval. Editorial and research by Gabriella Mayer, Andrew Foote, and Vlad Virtonen. Andrew Lipson and Max Fulton provided production support. Millie Shaw is director of audience development. The Open to Debate team also includes Gabrielle Iannucelli, Rachel Kemp, Linda Lee, and Devin Shermer. Damon Whittemore mixed this episode. Our theme music is by Alex Clement, and I’m John Donvan. We’ll see you next time on Open to Debate.