Where xMOOCs and Adaptive Analytics Both Fail (For Now)

No, this isn’t just an attempt to cram as many sexy keywords into one post title as possible. xMOOCs and adaptive analytics share an ambition: They both are at least partially motivated by a desire to teach at scale. With MOOCs, the goal is obvious. With adaptive analytics, less so, partly because there are multiple motivations and maybe because the desire for scale is not something that is polite to talk about due to a certain amount of discomfort with it. But the motivation is definitely there if you look closely, as I’ll get into in a bit.

The problem is that both of these approaches, in their current incarnations, miss one absolutely critical element of the teaching process. As you can probably guess from all the qualifiers I am using in my language, I don’t think it’s an inherent or permanent failing. But I worry that it’s a failing due to a deep cultural blind spot that we have about what education is, and that it therefore will be challenging to address.

The Case of the Missing Viola Player

There has been a lot of talk recently about the problem of Baumol’s Disease in education. William J. Baumol and his colleague William G. Bowen wrote about the problem of business sectors where costs go up but productivity doesn’t. They used the example of a Beethoven string quintet. You can’t play it with less than five musicians, but musician’s salaries go up because musicians always have the option to get a better paying job in another business. The music industry has to keep raising salaries at least enough to keep qualified musicians performing. So the cost of the performance continues to rise while the income per performance does not. Once a concert hall is full, it’s full. In that case, either ticket prices have to go up or the concert hall will go out of business.

Baumol’s Disease is an imperfect diagnosis of the cost problem in education because there is no evidence of wage inflation among faculty. The AAUP released a study this year that shows, among other things, that tuition at public two-year colleges have risen by nearly 45% above the rate of inflation over the past decade while faculty salaries have declined by 2.5 percent in real terms. The gap between change in tuition and change in faculty salary is even worse at four-year public universities. The smallest gap was at private four-year universities, but even there, tuition rose by about 29% above inflation while faculty salary increases ranged from 1.9 to 7.7 percent.[1]

Clearly, the aspect of Baumol’s Disease that people who are interested in education latch onto is the notion of the limit on productivity, which in this case is pretty much the same thing as the limitation on class size. One teacher can only teach so many students at a time. If you could increase the number of tuition-paying students, the theory goes, then rising income would keep up with rising costs and tuition wouldn’t have to go up. and if you can increase the number of tuition-paying students even more, then tuition could actually go down.

The first problem with this logic is that, as I just pointed out, faculty salary inflation does not appear to be the cause tuition inflation. It is possible that other scaling limitations would still make the analogy valid. For example, it may be that one of the principal causes of inflation is maintenance of the dormitories, classrooms, dining halls, etc., which, in turn, limit the number of tuition-paying students you can have. In that case, you could have a Baumol-like syndrome, by which I mean that there is a limit on productivity which is also the cause of a rise in cost. On the other hand, the cost increase could also be related to things like investment in research labs, upgrades to “student life” provisions (like better food in the dining halls or a climbing wall in the gym), or increased cost of administrative software. These problems would not be productivity-related.

But let’s leave that concern aside for the moment. Regardless of the ultimate cause of tuition inflation, it would be a Good Thing to figure out how to reach more students (cost-)effectively. And it’s certainly true that, regardless of the cause of tuition inflation, the ability to teach more students could be part of the solution. This is where MOOCs typically get invoked. It’s the whole “massive” thing. If you have 160,000 students in a class (as Sebastian Thrun and Peter Norvig did in their first course), then you don’t need to collect a lot of tuition per student at all to cover an awful lot of cost. Ten dollars per student gives you $1.6 million. From one class. We have solved the productivity problem. Hurrah!

Alas. If only it were that easy. There is a little bit of South Park underpants gnomes logic here:

The aspect of Baumol’s Disease that we tend to ignore is ironically the most obvious one: The reason that you can’t play a Beethoven quintet with four musicians is because there are five parts. While the cMOOC community has at least attempted to provide a theory of pedagogy to explain how they can achieve the “M” in “MOOC,” I have yet to see a serious account by any of the xMOOC providers to tell us how they think they can play a quintet with only four musicians. What happened to that viola player?

In Search of the Missing Melody

Let’s break the class experience down and see if we can find which parts transfer. First, there’s the content transmission. In the traditional model of the classroom, that’s lectures and readings. Readings scale. Traditionally, lectures have not. The biggest lesson that Sal Khan has taught the world is how incredibly cost-effective it can be to scale the lecture with the tools we have today, particularly if we don’t let ourselves get hung up on production values that have nothing to do with learning.

Second, there’s assessment. This is harder to scale. It’s only a little harder if you’re content with multiple choice questions, moderately harder if you want to see students actually solve problems that have cut-and-dry right answers, and a lot harder if you’re trying to assess the student’s thought process and writing skills. xMOOCs are definitely doing assessment. There’s an open question about how well they are doing assessment—I think it varies from subject to subject and class to class—but they are doing it.

Third, there’s remediation.


Almost every student gets stuck from time to time. Individual students will get stuck more or less often, at different places, over different things. When I think back on the best teachers I had when I was in school, the ones that leap to mind first and the ones who identified exactly what I was struggling with and helped me past the road block. “Remediation” isn’t quite the right word; the student could be trying to work through a problem that is far in advance of where the rest of the class is. The point is that the zone of proximal development is a real thing. There are places in every student’s learning path where she can learn things with help that she can’t learn without help. These places are individual to the student at that particular moment in the student’s intellectual growth. Therefore, one highly critical element of good teaching is being able to provide highly individualized attention to the student at critical moments.

This is very hard to scale for several reasons. First, by definition, you can’t just give the same thing to every student. Second, you need to be able to identify not only what a student is getting wrong on an assessment but why she is getting it wrong. The field of Intelligent Tutoring Systems (ITSs) has been giving this challenge a lot of attention for the last several decades now. I wrote about it in my post about LearnLab at Carnegie Mellon University. (The “cognitive tutors” I described are a subspecies of ITSs.) In Kurt VanLehn’s outstanding overview of best practices in ITS design, he describes them as having outer loops and inner loops. The outer loop is more or less the same that you get from any testing system. Did the student get the correct answer? The inner loop looks at the steps that the students took to get the answer and responds to them step-by-step, catching them at the moment they make a mistake and providing feedback that is directly relevant to that mistake. In other words, they do what a teacher or tutor would do.

Well-designed ITSs do work, but they are notoriously difficult to build, typically requiring both a great deal of expertise and a great deal of time. There are a couple of tools, like CTAT and Assistments, that attempt to lower the bar, but it is not yet clear that anyone has come close to cracking the nut of scaling and democratizing production of ITSs. And even if they do, ITSs are mostly proven for problems that require relatively clear-cut procedural solutions. Assessing and remediating the flaws in a student’s analysis in a history paper is another matter entirely.

So here’s an area in which a an entire sub-discipline dedicated to solving this problem is still struggling with how to scale production after many years of trying. Has the xMOOC contingent even addressed the remediation problem? A little bit, around the edges. At different times I have heard both Coursera’s Daphne Koller and Udacity’s Peter Norvig talk about sometimes being able to deduce where the student went wrong based on the specific wrong answer she gave. There’s something to that, but I’m not convinced that it’s more than an edge case that works from time to time for a handful of disciplines.

Update: I should say a word here about Coursera’s peer review tool, which does attempt to provide remediation as well as assessment. I think that calibrated peer review is an interesting approach that shows much promise for scaling remediation. But feedback on Coursera’s tool when used in actual courses has been mixed at best so far. Designing an effective peer review solution, with quality rubrics, the right incentives for students, etc., is hard, and there is a lot we don’t know yet about what works and what doesn’t. But to be fair, I should acknowledge Coursera’s significant effort to support remediation.

I’m not being entirely fair to the xMOOC providers. Later in this post, I will talk about an approach that some schools are using with xMOOCs to address the inner loop problem. But in all the MOOC mania, it’s astonishing that this largely unsolved problem is often being papered over. How is that possible?

The Viola Player Is Dead. But Who Killed Him?

If we are honest with ourselves, we have to admit that the viola part has been missing since long before MOOCs showed up on the scene—so long, in fact, that the music scores handed out to our young musicians as they take their roles in the faculty quinquartets are actually missing the part. Young instructors are never told that they need a viola player. They are never given any training in pedagogy. And they certainly aren’t given time or resources to provide individualized help. When I was in college in the 1980s, I sat in my share of 300-person lectures. Since then, the pressure to have larger classes and to teach more classes per semester has only risen, particularly in public colleges and universities. My guess is that college professor productivity has risen in the last decade, if all you mean by “productivity” is number of butts in classroom seats per teacher. The cost has been less time to respond to individual student needs.

In some ways I’m stretching the quintet analogy to the breaking point. The fact is that the lecture, sage-on-the-stage model of teaching is probably as old as college education itself, and maybe even older. This is one reason why I worry that fixing this problem as we attempt to scale will be difficult. It’s been a problem for so long that a lot of folks in academia can’t even see it as a problem. It’s just normal for them. And they’ve been aided and abetted in seeing the world this way by another industry player that has been around a lot longer than the xMOOC providers. I’m talking about the textbook industry.

As the time pressure on faculty increased, the textbook industry stepped in to solve their problems for them. Don’t have time to gather readings for your class? No problem; we’ll give you set of curated and editing readings that you can use out-of-the-box. Don’t have time to prepare lectures? No problem; we have slides that are consistent with the readings, notes on the slides, and questions you can ask your students during class. Don’t have time to make and grade tests? No problem; we’ll provide you with a test engine. Every time the pressure on faculty increased, the textbook industry was there to say, “Don’t worry about that part. We’ll do it for you.” A lot of what has shaped the textbook products in the U.S. higher education market over the last decade or more is essentially this unspoken agreement to hand off increasing amounts of the work involved with teaching to the publishers.

Let me be clear: I don’t see anything wrong or immoral about the idea of an industry that exists to solve teacher problems. The issue is that we haven’t had the right incentive structures to create an industry that exists to solve student problems. The solutions I listed above might solve student problems, but they are designed to solve teacher problems because teachers select the textbooks. And the main teacher problem that textbook publishers in the U.S. higher education market are paid to solve is increasing the throughput of the number of students seen by one faculty member in a semester.

This dynamic is beginning to change, for two reasons. First, students have become very clever about avoiding buying expensive textbooks, so publishers are having to work harder to create value that can’t be easily replaced by Wikipedia or Khan Academy (or the video lectures in a MOOC). Second, college and university funding is increasingly tied to outcomes, which means that a textbook publishers who can show that students learn better with their materials will be more likely to win the adoption. The transition is slow, but it’s happening. This is probably one reason for the blossoming of “adaptive learning” or “personalized learning” products. Unfortunately, I don’t see much evidence that many of these new products have an inner loop. It’s one thing to be able to say, “I see you’re still having troubles with fractions, so I’m not going to move you on to percentages yet.” It’s quite another to be able to say, “I see where you’re having trouble with fractions, so I’m going to give you instruction specific to the misconception you have about fractions that’s getting in your way.” There simply is no magic technology bullet to solve that problem yet. So I worry that, as good as the adaptive products may be at what they do well, we’re not going to notice that the music they are playing is simply not that missing viola part.

That said, all is not lost.

“It just so happens that your friend here is only mostly dead.”

Phil Hill recently pointed me to this video testimonial regarding San José State University’s experiment doing a flipped classroom using an edX course. (Sorry; it’s not embeddable. You’ll have to follow the link to see it.) As you might expect, the class basically used the MOOC for the lectures and assessments and did the remediation in a physical classroom. That’s the point of a flipped class. A traditional classroom forces the professor to put most of the work into the stuff that’s easy to scale and leaves little time for stuff that’s hard to scale. The flipped classroom…uh…flips that. When students run into problems, they are not alone. They are with an expert who can help them. They are also with their peers, who often can help them too. And a lot of the time, they don’t need help. In a lecture, the teacher spends the entire class period focusing equal amounts of attention on those students who need help and those who don’t. In a flipped classroom, the teacher spends most of her time with students who need her help and very little time with the students who don’t. This is a big adjustment for some instructors to make, but it can be done. In the SJSU video, I was particularly heartened to hear the professor speak so enthusiastically about his first experience with this kind of teaching and the new role he found himself in as coach rather than lecturer.

I don’t think it’s much of an exaggeration to say that the fate of higher education in the United States may depend on the ability of faculty to embrace this change. In the music of the classroom, the four parts that we are listening to today are increasingly playable by robots. If you look at this through the lens of disruptive innovation, the textbook publishers provide the…uh…textbook case of a potential disruptor. They have been working their way up the classroom food chain, progressively taking over the parts of the work that faculty don’t have time to do (or don’t want to do). MOOCs are rapidly covering similar ground. At some point, somebody is going to ask, “What do we need the instructor for?” Faculty could be replaceable in the quartet.

But that fifth part is a bitch for the robots. In my opinion, it also happens to be one of the most valuable aspects of the teaching process. I remember teachers I had who gave fantastic lectures. I also remember teachers who helped me to grow beyond what I could have gotten to by listening to lectures, no matter how fantastic they were. When I ask myself which of these two types of teachers had a bigger impact on my life, it’s not even a close contest. For the good of their students as well as their own career security, I very much hope that more faculty will pick up that viola and start practicing.

Share Button
  1. Thanks to Inside Higher Ed for the link to the study as well as a cogent summary. []

Google+ Comments

About Michael Feldstein

Michael Feldstein is co-Publisher of e-Literate, co-Producer of e-Literate TV, and Partner in MindWires Consulting. For more information, see his profile page.
This entry was posted in Pedagogy and tagged , , , , , , , , , , , . Bookmark the permalink.

14 Responses to Where xMOOCs and Adaptive Analytics Both Fail (For Now)

  1. This is a really good analysis. I think you correctly identify the missing bit – helping students over those rough patches.

    Interestingly, to my mind, although the problem of understanding and responding to a student question is an almost intractable problem for machines, it is generally pretty straightforward for humans. So what we have tried to do with cMOOCs is connect people with the humans they need to connect with to get over the rough patches.

    You don’t need an expert for this – you just needs someone who knows the answer to the problem. So we have attempted to scale by connecting people with many other students. Instructors are still there, for the tough and difficult problems. But students can help each other out, and are expected to do so.

    An example of what I mean: I just purchased a new xBox and a copy of MLB 2K12, which is a baseball simulator. My first effort to puitch saw me walk most of the batters, throw numerous wild pitches, and finally get out of the inning only after giving up 14 runs. The problem was, I didn’t know what to do; the MLB 2K12 instructions are far too vague, and if there’s in-game help,. I haven’t found it.

    I don’t need an expert in MLB 2K12 to show me how to pitch. I just need someone who knows what to do. Someone who can say “Well you move this control here then here and you’re trying to line this up with that.” Million s of people know the answer to this question. but I’m connected to none of them.

    Indeed, I don’t even need then to do the actual explaining. They simply need to recognize what my problem is, then point me to a video or instructions that outline the solution.

    Machines will eventually be able to do this, but they will first need to master natural language processing. This is going to take a while. In the meantime, if we want massive learning, we need o structure learning in such a way as to make asking questions easier, and as necessary, to provide more incentives to people to answer them.

    I don’t think the xMOOCs are ever going to do this, because their focus is on placing all the emphasis on the expertise of the instructor. To the extent that they respond to this need, they will become cMOOCs. But to the extent that cMOOCs become viable, the value proposition behind the elite universities is weakened. People don’t need experts; they just need someone who knows.

  2. Pingback: Where xMOOCs and Adaptive Analytics Both Fail (For Now) |e-Literate « Things I grab, motley collection

  3. Thanks, Stephen. Your comment reminded me that I should have addressed Coursera’s peer review tool in the post (which I have now done with an update). I agree that peer review is a critical component of scaling remediation. It’s hard to do reliably well—and there are some definite pros and cons to using a structured and artificially constrained peer review system like Coursera’s versus the more wide open approach in cMOOCs—but one way or another, we have to incorporate peer review as an integral part of our course designs to achieve scale in many cases.

    There is some very interesting (and early) research on when the quality of peer feedback can be as good as or better than expert feedback and when it can’t. At some point, I’ll dig into that literature and post on it.

  4. I am glad you mentioned 160,000 enrollments to AI . If you had said $ 10 at the beginning of AI course enrollment would be several thousands.
    MOOCs are not massive at all . At the beginning 100,000 or so finish is only 2.5 % or so .
    But still coming to cost and fee.
    Assume an online course enrollment is only 500 per quarter for 10 quarters
    at $ 200 then income collected 500 x 10 x 200 = $ 1,000,000
    But today unknown colleges charge $ 1,500-3,000 per course.Nobody says
    anything . EDX and Coursera are two different poles, south pole and north pole
    Please do not mention both as MOOC . EDX is non profits elite schools consortium, Coursera is a wonderful marketing company for profit

  5. Sorry. I want to learn from experts. Why should I be satisfied with someone just knows. That is the reason that MIT and Harvard are a brand name .
    They have the experts in their faculty even the best in the world . Otherwise they could not be so famous and expensive .

  6. You nailed it, the crux of the problem in higher education: “It’s [the traditional lecture model] been a problem for so long that a lot of folks in academia can’t even see it as a problem. It’s just normal for them.”

    Their [higher ed institutions and faculty] ‘normal’ needs to change, just as learning tools, methods and pedagogy has.

  7. Tom Abeles says:

    On target is the need to help (motivated) students over the humps. Having been involved in education, internationally, and from K-16 and above, the idea of peer help as suggested in the piece and the comments by Stephen Downes is that it works at a level where the problem is well defined (fractions or how to play a game). It still is inadequate in the areas where we have less defined and more nuanced learning. And it can be deadly in the field of medicine where peers may know an answer, even commonly accepted but lack the judgment, especially in “practice”. The move toward “competency” based measures attempts to get at this issue. Watching many list serves with professionals and having sat in on cMOOC’s, it can be most difficult to ascertain whom one can trust or to sort through the persiflage. Thus there is an advantage to being able to afford the expert. It may be one of the reasons that, in the US, we are seeing an increasing differential between those who can afford such access and those who can’t. It is eve more stark, internationally and not just in education.

  8. TOM
    You are right.
    We need some research to be done ” how to motivate students to learn and to make their living ? ” They are young, and just they do not care what will happen after 5 years .

  9. tom abeles says:

    I have an unsubstantiated belief that the problem starts at age zero or more realistically pre-school. I see this globally. By the time they get to secondary and post secondary the habit patterns are difficult to change because the students’ relationship with parents changes and the educational system at that point is not geared to engage sufficiently. Parents with the intellectual and social resources as well as financial assets, as the data seems to show, are, in the US, driving the gap between groups further apart. Again, walking across geopolitical boundaries or traversing across communities in the US, the differential is apparent. Strip away the rhetoric and “Closing the Gap” whether in education or in such areas as “micro-finance” has been spotty at best and has not lead to the hoped for translational tsunami.

    What “research” would you propose?

  10. For the assessment part in Math (and Chemistry), you might want to take a closer look at ALEKS (http://www.aleks.com/about_aleks/overview). It was originally developed from research at New York University and the University of California, Irvine. One of the salient features of ALEKS is that it uses artificial intelligence to map the details of each student’s knowledge through an efficient adaptive assessment. This frees the instructor from having to figure out on her own for each student exactly what they are ready to learn (which would otherwise be very time consuming and less accurate) so that she can focus on teaching them the most intricate material.

  11. Nicolas, I’m well aware of ALEKS, which also does not have an inner loop.

  12. Pingback: Re:Booting CA Higher Education: My First Thoughts |e-Literate

  13. Pingback: Making some “3100″ thinking explicit « The Weblog of (a) David Jones

  14. Pingback: A Taxonomy of Adaptive Analytics Strategies |e-Literate

Comments are closed.