The Gates Foundation recently pledged $1 billion over two years for AI efforts, with a large share going to education. Why the big AI push, even as Bill Gates warns of the technology's possible downsides for society? For this episode, Jeff talked with Patrick Methvin, interim president of U.S. programs at the Gates Foundation, who argues that the foundation can play a unique role as a bridge between educators and frontier AI labs. So what does the foundation see as the most effective uses of AI in education, and how is it guarding against unintended consequences?
The Gates Foundation recently pledged $1 billion over two years for AI efforts, with a large share going to education. Why the big AI push, even as Bill Gates warns of the technology's possible downsides for society? And what does the foundation see as the most effective uses of AI in education, and how is it guarding against unintended consequences?
Articles mentioned this episode:
“Gates Foundation Pledges $1 Billion to Combat A.I. Inequality,” in The New York Times.
“The turbulent AI era is here. The choices we make now are critical,” by Bill Gates.
“Can AI Improve Intro Courses? A New Courseware Project Hopes So,” in The Chronicle of Higher Education.
“One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment,” at the National Bureau of Economic Research.
Sponsor:
This episode is brought to you by Studiosity.
Learning is a human act of thinking, not simply the file a student hands in.
Studiosity champions the productive struggle. It happens in the moment when a student could easily outsource their work to AI, but decides to keep thinking instead. Studiosity provides the infrastructure to support that vital choice over and over. The work is supported, so students learn to think through it.
Keep the learning real. Visit studiosity.com/learningcurve. That's studiosity.com/learningcurve.
Jeff Young:
Hello, and welcome to Learning Curve, where we look at how education is adapting to the rise of generative AI.
I'm Jeff Young, a longtime education journalist.
Okay, everybody has heard of the Gates Foundation. It's given away billions of dollars to education projects since it was founded about 25 years ago.
And now the group is at this major turning point in its history. I actually feel like a lot of people didn't catch this news, but last year the foundation announced that it is starting to wind down operations, and that it's going to close up by 2045.
That might sound like the foundation is kind of fading out. That is the vibe I first got when I heard the phrase "wind down operations. But actually, leaders of the Gates Foundation say that they're rushing to spend the money more quickly, since they now don't have to worry about trying to make sure the organization sticks around indefinitely. In fact, the foundation is now doubling its spending.
And generative AI is actually a big part of the focus these days.
Just this month, the foundation pledged a billion dollars in grants toward AI in the next two years, with education making up a sizable chunk of that spending, so I have been curious to hear what the foundation's strategy is for AI in classrooms, and what kinds of AI projects that it's betting on.
And for this episode of Learning Curve, I got to ask the Gates Foundation directly.
Patrick Methvin:
Yes, I'm Patrick Methvin. I'm the interim president of U.S. programs at the Gates Foundation.
Jeff Young:
Patrick leads all the education work for the foundation, both K-12 and higher ed.
Patrick Methvin:
In the U.S.
We also have a global education program that is not under my remit, but we collaborate with them a lot.
Jeff Young:
Okay, so in the U.S. anyway.
So I was excited to talk with Patrick Methvin because I have so many questions about the philosophy that the Gates Foundation has when it comes to AI in education, and I'm also curious to hear what the foundation is doing to counter all the damage that educators say AI is causing to learning.
As listeners to this podcast know, we look at the good and the bad of AI when it comes to education.
And Bill Gates himself has recently started warning about the potential downsides for society when it comes to AI. Just this very morning, I heard Bill Gates interviewed on the Ezra Klein show, so he's kind of everywhere talking about this. In late August, Gates wrote a public letter saying that AI is creating one of the “most turbulent times in human history,” and that without careful implementation, AI could lead to mass disruptive job loss, and that bad actors could wreak havoc with this tech.
And he wrote that “AI could stunt our kids' development and replace human relationships.” (It made me think of our last episode about MIT's report about social learning.)
This letter by Bill Gates made clear, though, that he is not against AI.
Instead, it sounds like Bill Gates feels we're in this key moment in AI's development where leaders need to draw out the positives of the world-changing tech and work against potential harms.
So I started my conversation with Patrick Methvin by asking him just why the Gates Foundation is making such a big bet on AI in education.
AI is clearly a big part of the focus these days.
In fact, I just saw in the news that as we talk, it was just the other day that the Gates Foundation put out a release that it's committing a billion dollars in the next two years to fund AI efforts, equitable AI efforts, and 40% of that, as I understand it, is going to education. So AI, which of course is the interest in this podcast.
My first question is actually just to to really be broad and say why AI why is that a big focus of your education efforts as you guys enter this new period?
Patrick Methvin:
Yeah. Well, first, Jeff, thanks for having me on to chat about this.
Maybe I’ll start by putting that announcement that you mentioned in context, right?
Jeff Young:
Great.
Patrick Methvin:
So I'll put it in context financially, and then I'll put in context with a little bit of our what our strategy is because AI is a part of it, but it is not the strategy.
So $1 billion-that's a lot of money over two years.
That is less than 5% of the foundation spend during that time period, right?
We invest about $9 billion a year, and so that sounds like a massive, massive number. But some people read it as like you're going all in on that. I'm like, we're going all in with 5% of resources. So I just want to put put that in context, and it is a lot of resources, and I don't, I don't want to skirt that part.
In terms of U.S. programs, so a little bit of context on the strategy.
Yes, 2045 was a very focusing moment when Bill said we're going to, you know, wind down by then.
It was always meant to be a spin-down foundation. So whether that number was going to be 2045 or 2050. You know, you can say one versus the other, but it became a very focusing thing. It caused our leadership team to step back and say, "Wow, we've got 19 years. What kind of impact can we make during that time period?
And it brought us together in a very interesting way. We looked at you know for a learner who is going to be graduating in 2045, where are they now? And so it brought our K-12 work together with our post-secondary.
Say, how are we going to follow and support that learner all the way through? You think about, ‘What are their experiences they're going to have to have to meet what is our goal,’ which we'd mentioned was basically doubling the improvement rate of credentials, which would be about 10 million more credentials of value that would not happen otherwise.
Jeff Young:
It sounds like for the Gates Foundation, this is a moment to step back and review its broader goals and how the education work that it does can be coordinated toward this new foundation-wide goal of doubling the number of Americans who hold what it calls credentials of value. It wants to get that up to 10 million by the time it shuts down in 2045.
For this conversation, though, of course, my focus is AI, and so I pushed to try to find out which applications of AI the foundation sees as most promising for education.
Patrick Methvin:
So I'll go through some of what we call use cases. Maybe that's helpful. And I want to position this actually with something, Jeff, you just mentioned about the risks.
We constantly look at the benefit and risk trade-off right. And so let me start with one that I think we've talked about before that's it's a lot of benefit and not a lot of risk and then we'll hop around to other ones so I mentioned that I mentioned that credit transfer number, right?
So you got students who are transferring sometimes lose as much as 43% of their credits. What happens when you try to transfer credits right now? So a learner is at a community college, let's say, and they say, "Well, I want to go get a four year bachelor's or whatnot, and so they apply to that place. They send in their requests for credit transfers. And what then happens is either a dean or a faculty member or someone is sitting there looking at their old transcript and syllabus of this calc class and comparing it to theirs, maybe making judgments about what they think about the quality institution.
And two to three months later, they give an answer.
Now, in that time, the learner has already applied and probably decided where they're going, so they have no decision basically in, ‘Oh that's not going to work for me. I should go to this other place that will accept that credit.’
Okay, so this is how this works now.
AI is really good at pattern recognition, so what AI can do is to say, I'm going to take a look actually at those syllabi. I'm going to identify patterns. I'm going to make a recommendation that a human will ultimately decide on, which says I see X percent of overlap in what they studied. They showed these competencies were proven. Da da da da.
So two things can happen there.
One, you get an answer way faster. So a learner could take a look at that and say, ‘Oh, these five classes I'm not going to get credit for over here. So if I want a bachelor's at this place, I'm going to have to take these classes. It's going to take this much time, this much money, or this place accepted that and I fast tracked, etc.’ So it gives them a little bit agency. So time is one thing.
The second is quality. We're actually seeing with partners of ours like Ithaca and EQS very, very high alignment with what the human assessor was seeing, and so you're also seeing a quality angle there too. And it puts the kind of burden of proof about saying no.
It's like why would you not say no if you see? Why would you not say yes if you see this much match?
So that's one where not a lot of downside risk is frankly an administrative task. I'm not worried about cognitive offloading and all these things that are real worries. I'll talk about on these other things. That's just like we can make the system better.
Okay, so that's an example of a use case that I do not lose sleep over at night about the risks of AI.
If we follow through on some of these other things, so in the K-12 example, there's really two sides of this.
Yes, there is a question about, you know, a student tutor, and we're learning a lot on that. That's where you get into some of these things of people. If you're just using one of these things out of the box, out of AI, you could be completing your assignments faster, but getting lower grades on your test because it was not actually giving you the productive friction you needed.
So a lot of that work is how do you get productive friction out of that interaction?
I'll say like an example in our postsecondary work with Learnvia on that is they do have attached to their their what we call courseware. They have an AI tutor, and that AI tutor can see every interaction with the videos they're showing, et cetera, and at different phases, it acts differently.
So if it's early in your learning, you just watched a video about a concept, or you just went to class and heard about this, then it will scaffold things. But ultimately, we'll get you to an answer. If it's if you're if you're doing formative practice, then it's holding back more and more and more.
And of course, if you're on a quiz or something, it's not going to give that to you, so it's modifying its behavior based on the step of your learning, which becomes a pedagogical thing, right?
So there are things related to tutoring with the kind of example moving towards Osborne, but also gateway.
But some of the coolest stuff is actually for the faculty.
So we actually released in our Goalkeepers Report, which is a big annual report from the foundation, mostly related to the global sustainable development goals, we release an example of Kiddom Atlas where basically a faculty, a teacher, can see the results from like exit tickets and things like that, and it helps them situate the classroom the next day, saying like you've got a learning group over here, you've got a learning group over here.
That's not even exposed to the learner necessarily. That's actually for the teacher.
So some of the coolest use cases on these metrics are actually to the caring adult, or in some cases like the advisor.
Like I mentioned, we are working on some of the college and career advising as well as postsecondary advising. A lot of the organizations we're investing in, they're actually starting with guidance to the advisor first before they're going to a direct learner use case.
Now we know the direct learner use case is important because the questions that people are asking of these tools are happening like in college at like 11 o'clock at night. This is not during office hours. They're saying, ‘I need to know what courses I need to be paying attention to next year. I need to access this financial aid issue or whatever.’ No one's there to answer the call, and so that becomes like a very quick next thing we work on after we support the advisor.
So tutoring and faculty support, advising support, both for the advisor and the learner, the transfer work, and then I think the last thing that I would call out is the work with institutions where they are asking, "Hey, what use cases should I be paying attention to next?’
And so providing supports there.
And a lot of times, honestly, that's just surfacing good ideas from other institutions. We act as like an information broker because in in many cases, the innovation is happening out in front of us from just an innovative leader saying, "Hey, I wanted to do this with my back-office functions, or I wanted to do this with student success, and it's us just sharing with other people, creating forums, setting the table, which is one of the things that the foundation can do, which is around bringing people together.
Jeff Young:
Yeah, I'm curious. There's a lot of people talking about things like AI tutors and even some of the innovations you're talking about.
How do you see the Gates Foundation playing a unique role? How do you see yourself differentiated from other funders in the space, or what colleges might be able to do on their own, our educator education systems.
Patrick Methvin:
So, at the Gates Foundation, we look at a lot of our work with something that people refer to as comparative advantage.
So, it's not just like what are you good at, but what are uniquely good at that other people may not be good at, right, or have expertise.
And one of our big comparative advantages in this space is, and this is partially because of our history and because of our board and our board chair Bill. A, we have access to the technologist.
Jeff Young:
Yeah.
Patrick Methvin:
If you've been to an AI Labs office, you see it's frenetic. Like there's so much going on.
There's so much opportunity they're pursuing. They honestly don't have the time to slow down and engage.
Then and here's the second part: the communities that we're doing this to serve. So if we want to support 10 million incremental credentials, and we want to use this to close opportunity gaps by race and income and first-generation status, then we have a history of engaging with communities, building trust there, and then bringing them to the table. There's not a lot. There are some philanthropies that are super, super good, particularly local philanthropists, super super good at grassroots organizing. They understand the communities very very deeply, but they don't have access to the AI labs. And then there are some communities, some philanthropies even that are really really technology-focused, and they have access to that, but they don't have the other.
So I view part of our our role in this space as a bridge. We've been working for 20 plus years in communities, in middle schools, in high schools, in colleges, in. Community, we have super strong partners who have been working with institutions to improve for student success for years, and we have that bridge of access to AI and and a lot of technical acumen within staff here. Like we hire people who are at that nexus, and so I think that's a very specific role for the Gates Foundation at this moment in time.
Jeff Young:
Is there an example of where you see that kind of like the fact that you can, you know, people at your at the foundation can be you know kind of in the thick of the AI revolution and be connected to the education world in the way you described.
Patrick Methvin:
Yeah, let me give a very concrete example.
One of our portfolios I mentioned is the college and career advising space. So you know the tools that learners are using, and sometimes as early as middle school, but high school. We have for years invested in organizations that have done that kind of on an analog basis, like the One Goals of the World or things like that. Great organizations, but hard to scale, right? Great impacts. You can see it, particularly if the high school sets aside time for that in their day, amazing, amazing. We have been asked by many of those organizations, is like, what are the technologies that we should be using to underpin our human to human type model?
A few years ago, we hired a guy named Tim Herron into the foundation. Now, Tim had built an organization in Tacoma, Washington, that was specifically focused on that kind of advising process, and it was very much human to human. And he built it, served 1,000s of students, amazing stuff.
Well, as it turns out, Tim also was a computer science undergrad major and one of those closet tinkerers, right? Like he had a server at his house, and he's playing around with this.
And so, when the new technology was coming out, he had his own specific curiosity, and so he was like tracking on this stuff. And so, we had put him in the position to lead this team. So, what then happened very rapidly? He could sit down with the folks at Anthropic, who's a partner of ours, and he could bring context to where the models were struggling, and saying, "Well, this is actually what happens in that engagement between an advisor and a buy Z XXX. This is what we have to solve for.
It's like either this is an information problem, or this is an engagement problem, or this is an affect problem. He personally could bridge that.
Then he could also engage.
We have since engaged with Career Village, which has all kinds of different information about career tracks. Pretty good technical acumen on that team, and he could bridge that.
And by the way, when he was working previously at his other gig, he was engaging with folks from Microsoft on a regular basis, and so he could say, "Well, what if you could put something like this on your platform?
So that's like at a very human level an example of the kinds of talent that I think we can bring in, and then that means they are networked with each of those respective communities.
So I could probably say the same thing about our K-12 tutoring, our postsecondary work in courseware or advising or many others,
I do think that that's something that we have to offer in the field.
Jeff Young:
So it's not just cash, which people do need and use these to support these, you know, like the the work.
So, over the 25 years or so that the Gates Foundation has existed, I mean, the focus has been on evidence-based solutions. But by nature, these AI approaches are often untested. Even though, of course, you are testing them, but it's so new.
I guess why bet so much? As you say, you're doing plenty of that's not AI, but you are doing a lot of AI.
So why not just put that money into more tested solutions, like you know, why not just put the same amount of dollars into like scholarships or or improving student teacher ratios or things that that kind of have known to work.
Patrick Methvin:
Yeah, so maybe two two things underneath that related to the examples you give things like improving student teacher or student advisor ratios, it may sound like we have a lot of resources, but we are a drop in the bucket compared to overall education spend. We cannot fund every district to add five advisors, and like that is the role of policy.
That is the role of policymakers responding to taxpayers. Now we can surface that evidence to say, "Hey, you'd be doing a lot better if this wasn't a 400-1 ratio. But we don't have the funds to do that, despite having significant resources-not even close.
And so, back to this kind of comparative advantage, what what can we do so we can surface the evidence behind some of those things that you mentioned, Jeff, and that can be some of our policy advocacy work to say, hey, we should really be pushing for this, or or hey, you know, one of the reasons why FAFSA hit a a record high last year was because you're seeing policies in states about FAFSA completion to graduate. So we can absolutely do that, and we will continue. You to do that now on the AI space. What I want to call out kind of a pattern for how we go about this because typically our role isn't like throw some AI solution spaghetti against the wall and then see if it works. There's a process, right?
So the process starts very analog. So it says, what do we know about what good looks like in that space? So I'll stay with that pathways advising example, just because we we touched on the first thing that we did is we've learned from years of working on the analog basis, and we funded organizations like Cara, which represents good college advising, to say give us a definition of what good looks like, and then we say turn that into a rubric. Now that basically becomes a version of a benchmark. Then we can go and say, oh, let's take a look. How are these models doing against that? And you can score it. So before we start talking about solutions, where are we? Where is the state of the field? Is it good? Is it bad?
Does this like tool out of the box just like knock it out of the park. No, the answer is almost always no. And by the way, as each AI model has progressed from you know version two to three to four on things like the K 12 tutoring use case, it has not improved on those benchmarks. So it's improving on all of these things like all over the board, but not right.
Jeff Young:
It's like solving the Millennium Prize problems in math, but not necessarily tutoring getting better.
Patrick Methvin:
Very little movement on tutoring, right?
And so that's the first step: is understand like where are we? What is state of the art within the field?
And then the second step is what are the things that experts would say could make that better?
And so that's where you take the benchmark and turn it into an understanding of like, huh? This is interesting.
It's constantly giving advice to go down this path, and it's not clearly looking at these data sources, or maybe its tone is wrong in these steps. Right. This is where you have to bring the subject matter experts, in this case, advisors, and into the fold and say, tell us about this. Like literally looking at transcripts of interaction and annotating, right?
So that's the second step, and then you can build what we call public notes. Which for, you know, the AI geeks following this, that could be a model harness, which says we're going to attach this to a model, and it will instruct it to do the following things, or it will point to the following data sets automatically, right?
And then you do the process again. You say, okay, well now let's take a look with the model in this harness and put it up against that same benchmark we used. Did it get any better?
Usually, you see some kind of improvement, but it's still not human level quality, right? So there's a version of that which is wash, rinse, repeat.
Along the way, you also have to be looking at people who are playing around with this, experimenting with this at a very very small scale, and saying which parts of this would you rely on the technology to do versus where do you need the technology to say go talk to a human, and how can you and so that part of R&D is implementation R&D that you can only do when you have real people engaging. Usually, we try a step before that, which is where an advisor will role play either with a model or they'll role play with another advisor acting as a student before you put it into a live environment. So you can see there's like different levels of tests which represent risk before you say, okay, now let's put it out there.
Jeff Young:
His point is that the Gates Foundation is doing its homework before it recommends anything in education, and that maybe goes especially for AI.
Patrick Methvin:
And I think that's part of having a clear AI safety perspective, which, you know, you're hearing the labs talk about it. You're hearing you know you really if you're if you're gonna be thoughtful about this, you have to have perspectives on that.
So we invest in folks like Common Sense Media and others who are developing versions of this to advise then also us. So then we can translate that into okay, what's the playbook if you're an investment maker? What things will you do? What will you not do? At what phase will you release this?
Like it's it's setting those steps because candidly, Jeff, there are not policies for that, right? Like if I'm in if I'm in health, I know there are specific rules to advance something through a clinical trial and all that. They're very clear, concrete rules.
That does not exist in education, and frankly, that's not an AI problem. That was a problem with general ed tech even before machine learning. Just in the think about the adaptive technology days, there was still no rules about that. That's not an AI issue. That’s something that we should have been working on. And so, if philanthropies are not building that in for their own accountability, they're just feeding into the problem.
And so, I you know I don't know if that fully answers your question, but that's kind of how we regimentedly try to step through. Have we made mistakes on that? Will we make mistakes? I'm sure we will, but we're trying to work with our staff of, like, here's the process we do before we put anything out into the world on that.
Jeff Young:
After the break, what has the Gates Foundation learned in its previous efforts in ed tech over the years that it's bringing to its work in AI?
Stay with us.
Sponsor:
This episode is brought to you by Studiosity.
Policy dictates one reality, but the campus lives another. Studiosity provides the essential infrastructure to defensively validate the genuine learning journey, moving the sector towards a model of support and validate by replacing vague AI scores with binary clarity, Studiosity empowers students to stand up for their own work and self-validate against a personal baseline. This builds persistent authorship continuity, shifting students from passive suspects into active owners of their own thinking.
Simultaneously, Studiosity enables educators to set clear expectations using the AI assessment scale, giving them back control over assessment security and replacing detective fatigue with a richer story of the cognitive learning process.
After all, learning is a human act of thinking. Keep the learning real.
Visit studiosity.com/learningcurve for more details.
Jeff Young:
Hey everybody! I want to remind you also of another project that I am working on these days. It's my newsletter on AI on campuses for the Chronicle of Higher Education. It's called Jagged Intelligence, named after a term that researchers came up with to describe how AI is so surprisingly effective at some tasks, but has major fails in others. What do you do with a tech like that? I hope you'll check out Jagged Intelligence and participate in our regular flash polls to share your thoughts and concerns about AI. Just sign up at Chronicle.com/newsletters.
Now back to the episode.
At the college level, the biggest single bet that the Gates Foundation is making in AI is in a new nonprofit called Learnvia, which launched out of Carnegie Mellon University.
I've been covering that universities work in learning science and applying it to online education for years, and the Gates Foundation put in $55 million so far in this Learn Via project, one reason I'm mentioning this is because you'll hear Patrick Methven talk about this project a lot. Learn Via offers courseware and an AI tutor focused on gateway math courses, those intro classes that can trip up so many students, and yet they're crucial to entering STEM fields and opening opportunity.
My colleagues at the Chronicle of Higher Education recently reported that professors from 38 colleges are participating in a Learnvia pilot that's focused on Calculus I.
But as the Gates Foundation supports that kind of effort, I'm wondering if it's also looking carefully for signs that the AI uses aren't performing as hoped, or that they may even be counterproductive. After all, there is that shadow of social harms to learning that Bill Gates himself is talking about. Is there anything you're looking for or two as a foundation where you might say, you know what, this isn't proving to be? Is there some metric you might see or have on the list that you're looking for that might say like maybe this isn't the right tool for education? We might pivot, or not, or are you finding that you feel like it has that potential down the road?
Patrick Methvin:
Yeah. So a few angles to that question that I think are they're important. One is on you know what are we looking for? Ultimately, we're looking for performance and improvement in learning, and that was the same with before AI with the technology stuff as well. So we set these goals and benchmarks based on two things, and I'll take the K 12 example, but I could do the same with post secondary. Our goal is 1.6 years of acceleration in one year for learners who are behind grade standard.
Jeff Young:
Okay,
Patrick Methvin:
And the reason for that is because if you go look out in the field right now, you're going to see 6.7 million students who are over a full grade behind in math, which means they won't even get to that starting line of Algebra I by ninth grade unless they accelerate.
Okay, so current tools are usually meant to just kind of get you to the grade level. But what do you do if someone's behind grade level?
You actually have to accelerate, and that's where something that is more dynamic in its interaction, like AI, is better than kind of a rules-based program type tool.
Now, why 1.6?
Part of that's the math of, like, what would we need to get more learners there.
The other is that's what the best human tutors can do. 1.6 years of acceleration.
So, if you have the means and you can get a personal tutor, tutor, that's what they can deliver. So, that is the aspiration.
The second thing we look at is what are the tools right now getting us, and it's all over the map because I think as you will appreciate, the tool alone doesn't get it done. It's the tool plus the implementation, right?
And so then we take a look at that and say, ‘Hey, some of the better tools out there with like aligned curriculum and good pedagogy and stuff, maybe they'll get you 1.1 or 1.2, and if you put it in the right setting with the right, you know, teacher or faculty member, maybe you can get to 1.3 or whatever, but not 1.6.’
So when you say, ‘What do we look for to say this isn't working?’
You know, we're going to run at that target and that goal as hard as we can for you know X periods of years. And if it doesn't work, you know, well, one, if it falls short, it gets 1.5.
God forbid, like that's great, but we set we set goals related to that.
The second thing I would say is we do pour a lot into evaluation of both the learning science that's going into it as well as the technology itself.
So the reason why we made a big bet in postsecondary on starting Learnvia, which is with Carnegie Mellon University, is because if you were to make a list with a Venn diagram of institutions who have a deep history of learning engineering and learning science and AI acumen, there's not a lot of universities that are going to fit in that Venn diagram like Carnegie Mellon University.
So part of that is an acknowledgement. We're going to need to keep learning about each of those things and each of those things together. And so that's, I think, the second piece is constantly learning about that.
The last thing that I would say there about that evaluation is you have to be super thoughtful, not just about evaluating the tutoring moment, but what happens to that learner?
How much do they retain to the next class or the next class?
And this is the thing. This is actually why the AI lab.
Well, this is one of the reasons why the AI labs first went to tutoring and not advising, because advising is a multi-year, longitudinal engagement with a learner without a single right answer. Your answer may be different than my answer, right? That's super hard for the way the AI folks think, which is, like, create a benchmark. There's a right or wrong. We can hill climb based on that. That's super super hard.
I would actually say something similar applies in tutoring, except in the near term, you can look at a benchmark that says, "Did you help a learner get to that, etc. What you don't know is afterwards how deep was that learning, and so we actually have to set up evaluation protocols where we can study, longitudinally, what is that doing to the learner? Is it improving that deep learning, which we believe comes from that productive friction?
So those are a couple of the ways that we, you know, we think about it and approach a problem.
Jeff Young:
Yeah, and I guess there are these other factors that we like the cognitive offloading and the concerns about.
Yes, these tools are designed, as you say, in this beautiful use case of like guiding learners, and but once you get into high school and college, especially when there's more agency by the students, they're just they're able to go use the tools that that the major companies are offering AI, like Microsoft and others, are that they often kind of go around and maybe use those tools to instead of learning on their own and doing the productive friction.
And so I guess at what point, yeah, I guess do you have any sense of how to contribute as a foundation and toward applying these so it's not displacing learning by encouraging AI and as a whole like that having more side effects than benefits.
Patrick Methvin:
Two things that we think about there.
One, I think the field of assessment is going to change a lot in the next five years.
Jeff Young:
So, like figuring out what the student knows, yeah.
Patrick Methvin:
And there's both headwinds and tailwinds on this, right?
Because so the headwinds are everything you just mentioned, Jeff, which is like, well, if I'm doing a take-home test, and okay, even if your platform's locked down, right?
Like our Learnvia tutor that we fund, like you can only ask it certain things within that tutor, but they could have another browser open right next to it, or their phone, or whatever.
Jeff Young:
Yeah.
Patrick Methvin:
So we're going to need a lot of innovation on assessment because of the headwinds, the risks on assessment.
I think you're going to see more and more, you know, technologies that are and proctoring technologies that are coming out to ensure that.
But there's also interesting tailwinds, so that we need to take advantage of.
So examples of that AI, and I'm seeing this with folks like WGU can through multimodal can observe. They can observe things like if you want to talk out a problem, or shoot if you want to talk about things like problem solving approaches or some of the durable skills, you can literally record a team exercise and what.
Jeff Young:
This is Western Governors University, which does a lot of online and yeah…
Patrick Methvin:
Exactly. And there are others doing.
They're not the only, but there are others doing this. But if you want to get into some of the skills that the employers are saying that they're hiring for, which we could get into an argument about whether skills-based hiring is real or hype, it is increasing to some degree. I do know to the degree they're talking about open question.
But what some of these organizations can do is to say, ‘Well, you're looking for teamwork skills, or you're looking for this, et cetera.’ You could literally videotape, live, a group of people solving a task, and it can annotate those things. So it could tee up to a faculty member. Here's the observations of the behaviors that we saw that align to this or that.
Ultimately, you'd want for assessment and grading, you'd want the faculty member to say, ‘Okay, well, here's what I do with that.’
But it opens up new ways to understand what people learn, and that would have taken those things, particularly observational, take hours and hours and hours, super expensive.
Now you can get kind of a head start on some of that, which it could take you to individual clips of like, ‘Hey, here's an instance where this happened, or here's an instance where this happened.
So I think assessments got changed a lot, both to mitigate risk and because I think it opens up new opportunities for us. And that's AI-enabled assessment. These were things that we could not previously do with assessment. So that I think is an area of excitement for me.
Jeff Young:
Sure, sure.
Well, you know, we are at a time with technology where you know we have this rushed Russian AI and a lot of excitement and some concerns.
But we also at schools we have this kind of very different kind of conversation suddenly around ed tech and and kind of some schools kind of looking to take away, you know, take computers and and screens out of classrooms more, and maybe a sense of like, did we go too far with inserting technology in, and do we need to recalibrate that?
So I'm curious, you know, the Gates Foundation was involved with a lot of those efforts, and plenty of other institutions as well. What do you think the foundation has learned from the internet era and the previous, you know, kind of innovations in ed tech that you're bringing to this moment of AI revolution in education?
Patrick Methvin:
Good question. There's a few things there.
One, first, I think I want to acknowledge a pretty deep conflation of a lot of things going on.
This is more on K 12, and it's the screen time and AI and tech. Like these things are all being lumped together.
Sure. And I do believe there when we talk about AI safety, there needs to be expectations developmentally, right?
And so this is where I agree with some of the even some of the conflations between screen time and AI are fine on this. To say, should a kindergartner have as much access as I don't know a college sophomore? Probably, probably not, right?
So we need to get better on our research science around what is appropriate developmentally at different ages. So that's the first thing. I just want to acknowledge some of that stuff is getting conflated.
The second, and I know this is like a duh kind of insight over 20 years, but it really is what does the implementation look like? And so I'll give an example with Learnvia with our postsecondary courseware with Carnegie Mellon, you see a faculty member who has a 200 person calculus section, right? They are not doing individualized teaching, generally seeking in that lecture hall. They're probably not even leveraging high impact practices or evidence based teaching practices.
Jeff Young:
Okay,
Patrick Methvin:
That's before AI or technology or whatever the case may be.
Jeff Young:
The kind of a human the human teacher scenario.
Patrick Methvin:
You get you just get ‘sage on the stage’ talking to folks and say, ‘Hey, read this thing in your book later, etc.’ Now, what we're seeing from the greatest implementations is the coursework is actually helping faculty flip their classrooms.
So people have been talking about flipped classrooms for how long? Yeah,
Jeff Young:
It's been a minute. Yeah, a little while.
Jeff Young:
A quick definition for those who don't know this term: flipped classroom is an approach that a lot of college teaching experts have been pushing, where the professor makes a lecture video and assigns that to students to watch for homework.
So when they get together in a physical classroom, they can use the time for more active work than just sitting and listening to a professor talk. There's a lot of research showing it's a more effective way to teach.
Patrick Methvin:
But now they've got a tool to do that because we embed faculty professional development in the courseware to say, hey, if you do the following, assign this short video and these trial things before class.
Then when you get to class, break out the groups in the following ways, and then roam around with you and your TAs, roam around and pop into the live action that is happening, and then now you start. Knocking off some of the evidence high evidence based teaching practices that they probably would have loved to do anyway.
So I'd say the the first and most important part is understanding how do you insert this.
And so in the Khan example, that's you know Sal talks a lot about dosage. It's like I can't get more than the 5% problem. I can't get more than 5% of this students using this to the degree that shows you will get impact, right? I think it's more than just dosage; it's dosage and how it is used.
And so, what we try to do with the technology is set up the components that help on the faculty practice. So here's another example: we've partnered with this group called the Equity Accelerator that looks at things like issues of belonging and things that we know matter to student success.
So inside the Learnvia courseware, they have a quick quiz that will pop up every once more, or survey type thing. Ask two or three questions about belonging, about confidence, etc.
They don't have to go. It's not a separate thing. The faculty doesn't send member doesn't send it out there. It literally pops up there.
Now here's the second part: is that information goes into the faculty's view, so they can see going into the next class. Oh wow, there's portions of your student population who just don't feel that this makes sense for them, or this is relevant to them, or whatever. Here's three ideas that you could try in your classroom to do that.
So I'd say one of the biggest aha's, Jeff, is like what is the R and D about implementation that should both inform the product, but then also the faculty practice. Again, I think a lot of people would say "duh" to that one, but are they infusing it in their work? And I think that's a super important piece that we're trying to do. I could say the same thing for about our advising work, about our K 12 teaching and learning work. But I think it's critical.
Jeff Young:
Yeah, I think it's funny because what I'm thinking as I hear that example is the culture of teaching, and really these courseware implementations at Carnegie Mellon.
You know, even before you were funding them as heavily as you are now, there they've hit upon you know it's like the the it's trying to change that stage on the stage to guide on the side and all the things that people have been talking about and flip classrooms. There are these ideas that are out there, but they they kind of go against the culture of the way teaching has been done, and and I do wonder you know whether even if I were a teacher, whether it's like how effective or how realistic is it to have these these you know prompts coming up to be like do this do that, I think it sounds interesting, but I can see I can see the challenges of of implementation that that are just kind of real when you have these the culture of teaching meets these these new technologies at scale.
Patrick Methvin:
It's super hard. I mean, it's you know people talk about change management is tough, and it's I mean it's super hard if you've been doing something in a certain way for 510, 15 years. That's not that's not the easiest thing to do to change practice. I what I hope, and what I've seen, and what will be amazing about this moment, in the best of scenarios, is it asks people to interrogate pedagogy and what is it that they do as humans that drives deep learning, and even asking that question, I mean, you and I both know the percentage of faculty who are trained in teaching is very low. We've seen some advances in that, like over the past 10 to 15, years.
Jeff Young:
Sure, it hasn't been the culture at a lot of universities, right?
Patrick Methvin:
It's very low, and so giving people the opportunity to reflect on that and say, "Wow, if I have a marginal hour, where would I spend it? Like, what is the most impact? Is it to set aside office hours for initial time. Is it that in the classroom I'm going to go make personal contact with someone to to motivate them?
I mean, how many times do we have to be in sessions where you ask, "Tell me about your favorite teacher ever, and then what was it about that? It was usually not about they were really good at helping me factor polynomials.
Jeff Young:
Yeah, not that they followed pedagogical practice.
Patrick Methvin:
Yeah, it is. I felt motivated by them. I felt seen. They showed me that I could do this. Right. It's those sorts of things that are uniquely human. That I hope we come out of this wherever we land in all this AI stuff.
I hope we come out of that with an excuse to interrogate our pedagogical practices and how we can help learners the most.
Jeff Young:
Yeah, and I guess the question I was kind of getting at earlier too is like, when do you know if it's getting in the way too much versus when it's actually an enhancement?
Patrick Methvin:
Yeah, I mean, this is where you got to put your money where your mouth is on the research front.
You know, I mentioned the example before about multiple iterations of the LLMs not actually helping on the education side, and we've seen that with some of our ed tech part.
Actually, I think Khan is one of those that studied that too. They said we swapped in the latest model; it didn't actually help at all. In fact, it was giving him answers away when we'd already put a harness on to say, "Don't do this.’ Right. Right.
So I. I mean that that's one example. I also think that there's just like some deep, deep work to be done about what is uniquely human and what is most motivating out of some of these interactions.
And I'm surprised sometimes. Like I remember early days seeing some of the learner preference work where they were just using out of the box models for tutoring and whatnot, and they said I actually like this better than my yeah some courses have like a chat going on the side. Sure, I feel embarrassed by that, and I'm like, huh, that's interesting. I don't know if I would have thought about that, and they preferred for certain types of questions to ask their AI.
And this is before this is a few years ago. This is before we've even tuned these things. They just said, "I want to get a first blush question from that.”
Now, the issue from that is it may respond without some of those pedagogical guardrails. But the notion of like, ‘I feel embarrassed or I don't want people to know. Okay, so let's run with that. And if AI is getting in the way of that going the opposite direction, then we need to solve for that, and then you take a look.
Is it practicing the right kind of pedagogical practices, not just giving away answers, creating a little bit of friction there? I think that's a pretty critical question as well.
Jeff Young:
I want to to shift gears a little bit and ask about your role because you know there have been some staff changes at Gates lately.
You're the interim head of education efforts there. You, I guess, do you? And you've just had this big strategy in education that we talked about a little bit already, and that that is driving that was announced pretty recently as you were kind of coming into your interim role. Do you expect the foundation to move in some other new big strategy in the next year or so?
Should people expect more changes to come or another shift in the near future?
Patrick Methvin:
I don't think so for a few reasons.
One, this new strategy was not something my predecessor Alan Golston just cooked up in his office, and it was like, here it is. Like this was a collaborative effort with the entire U.S. leadership team. So, to the degree that you see continuity there, there's ownership there. Everyone is excited about that work. This is what we want to be doing. So that's the first thing.
The second thing, then you could say, well, what if you bring in a new president? Doesn't that change the equation. The way our foundation works with our kind of delegated authority from our board and our CEO is they are very bought in to this, and by they I mean Mark Suzman, our CEO, and Bill, our board chair. And so it's not like they're just going to change their mind on some of this stuff too.
And in fact, in the search process, we are looking for people with full clarity about what the strategy is, and so for those two reasons, I don't expect you know dramatic changes, Jeff, in the near term to our work.
What I will also say is that this field is moving rapidly, and so that doesn't mean we're going to be stuck with no changes for the next 10 years. Like we're going to learn and adapt, but not for structural things like you know changes in leadership, et cetera. And I think you know for my appointment and the interim basis, at a minimum, they chose someone who's been here for 13 years, who has deep relationships with our board chair and our CEO and many of our communities, and so it-it was like let's let's keep moving in this direction.
Jeff Young:
In sort of wrapping up and thinking through it, I what do you hope then, especially with AI, since that's the focus on the podcast?
Is sort of where do see as your positive vision of where you hope in five years that things are really different in the way teaching is happening because AI is in there.
Patrick Methvin:
I would say one that we're removing as much unproductive friction as possible.
So the credit transfer would be an example of that, or even you know navigating your homework tools or whatever at night, right? Or getting basic answers of like when's my next quiz?
That is not a friction that's helping anybody. It's not like you know 20 years later you're at the job and you're like I'm really glad I couldn't find that on the LMS. Right, so that is removing unproductive friction, which is also an efficiency play for learners and and institutions, and that we're finding ways to amp up the productive friction mostly through a combination of as it's with the tool, how is it motivating the learner to proceed, and then second, how does it free up faculty members to do things that are uniquely human, which I think will increase the productive friction because you have a student who is motivated, engaging, and persisting.
So I would love to see that if that is happening in and out of the classrooms.
You're going to see more students persist. You're going to see them following the paths that they would hope to follow because they don't get shoved out of a field of study that they thought they could be interested in because they had a really bad experience in a classroom, and they will move on to completion of a credential value and hopefully family sustain.
So that's what I hope we would see.
I would also hope, in terms of the ecosystem, that through our collective work we get the attention of the AI labs to pay attention to this, even when they're training their core models back to the point where we've seen huge advances, but not so much in the education use cases. My hope is that when they're training next generations of models, they're bringing in data points and approaches that can identify. Oh my gosh, someone who just came to this thing randomly for kind of a tutoring use case, I can now flip to from just like here's the answer to let's go on this learning journey.
Like I would love for that to be almost natural for those. I don't think that's going to happen as easily, which is why I do think there's very much still a space for the ed tech vertical, and it's not just going to be all AI labs all the time. I think there's there's going to be a happy medium there.
Jeff Young:
It's so interesting. Well, thank you so much for taking the time today. This is really interesting.
Patrick Methvin:
Awesome, thanks, Jeff. It's a pleasure.
Jeff Young:
As I put this episode together and reflected on the conversation, the biggest thing I keep coming back to is this comment Patrick made that AI tutors have not worked as well out of the box as experts had hoped. In fact, one AI tutoring project that the Gates Foundation has been funding is an AI tutor by the nonprofit Khan Academy, and Sal Khan, who leads that group, recently said in interviews that these AI tutors just have not been as successful as he imagined.
There was a study that came out last month of some middle schools in Tennessee that found that students used the tutor infrequently, and when they did, rarely engaged it in substantive mathematical dialog.
But this podcast episode shows that that doesn't mean that Khan Academy or the Gates Foundation has given up on the idea of AI tutoring.
The promise is still that AI can solve social inequities by unlocking education as never before. But I am seeing more evidence that AI tutoring is just harder than people thought, and I was curious about this idea in this interview that it might just take more serious attention by the AI labs, like OpenAI, Anthropic, Google, to actually bring the promise of AI tutoring to reality. Can AI tutors get as good or better than humans?
This is one of the big issues we're watching on the podcast, so we'll keep digging in to why AI tutoring has not been more effective.
If you have thoughts or some data or an opinion on this or other topics, send it my way to Jeff at LearningCurve.fm.
This has been Learning Curve.
Every episode, we look at this dramatic moment in education as it struggles to adapt to AI. Please follow the podcast wherever you're listening, so that you don't miss any episodes. We're gonna have a lot of good ones this season, and tell a friend about the show on social media, or in person if you're at a conference or maybe just in the hallway.
This episode was put together by me, Jeff Young. You can find links to related material and more about my work at LearningCurve.fm.
The theme music for this episode is by Komiku, with additional music by Blue Dot Sessions. Episode art was generated by Midjourney.
We'll be back in two weeks with another episode.
Until then, thanks for listening.