Can Pearson Solve the Rubric’s Cube?

Love ’em or hate ’em, it’s hard to dispute that Pearson has an outsized impact on education in America. This huge company—they have a stock market valuation of $18 billion—touches all levels from kindergarten through career education, providing textbooks, homework platforms, high-stakes testing, and even helping to design entire online degree programs. So when they announce a major change in their corporate strategy, it is consequential.

That is one reason why I think that most everybody who is motivated to read this blog on a regular basis will also find it worthwhile to read Pearson’s startling publication, “The Incomplete Guide to Delivering Learning Outcomes” and, more generally, peruse their new efficacy web site. One of our goals for e-Literate is to explain what the industry is doing, why, and what it might mean for education. Finding the answers to these questions is often an exercise in reading the tea leaves, as Phil ably demonstrated in his recent posts on the Udacity/SJSU pilot and the layoffs at Desire2Learn. But this time is different. In all my years of covering the ed tech industry, I have never seen a company be so explicit and detailed about their strategy as Pearson is being now with their efficacy publications. Yes, there is plenty of marketing speak here. But there is also quite a bit about what they are actually doing as a company internally—details about pilots and quality reviews and hiring processes and M&A criteria. These are the gears that make a company go. The changes that Pearson is making in these areas are the best clues we can possibly have as to what the company really means when they say that they want efficacy to be at the core of their business going forward. And they have published this information for all the world to see.

These now-public details suggest a hugely ambitious change effort within the company. Phil and I have consulted for a few textbook publishers, including Pearson, and I worked for Cengage for a year and a half. We have a pretty good idea of the magnitude of the change management challenges these companies face right now and the strategies that various publishers are bringing to bear in an effort to meet them. I can say with absolute conviction that what Pearson has announced is no half-hearted attempt or PR window dressing, and I can say with equal conviction that what they are attempting will be enormously difficult to pull off. They are not screwing around. Whatever happens going forward, Pearson is likely to be a business school case study for the ages.

As if all of this drama weren’t enough, Pearson’s strategy raises another question which should be fascinating for educators; namely, can a rubric transform a multi-billion-dollar company?

Fair warning: This post is ridiculously long. Even by my standards.

Transforming U

Before we can really dig into the questions at hand, let’s talk about people’s beliefs about Pearson and how those beliefs may color their interpretations of the company’s actions. I’ve been thinking a lot lately about motivated cognition—the depressingly robust and pervasive process by which humans strongly tend to draw conclusions from new facts or arguments that are consistent with what they already believe (and want to believe), particularly about hot-button issues like politics. (As usual, Mike Caulfield has already written a thought-provoking post on this topic and some of its implications for education.) Now that Blackboard isn’t suing anybody anymore, Pearson is probably the single most emotionally charged brand in the education industry. Anything at all having to do with their actions or intentions is likely to function as something of a Rorschach test. It can be helpful in such situations to try to analyze the questions before us in a different, less emotionally charged context first, and then apply what we know about the context afterward to see how it might change our judgments.

To that end, I propose a thought experiment. Imagine if the president of U.C. Berkeley wrote the following:

The learning challenge is unrelenting in scale (by 2025 the world’s population will have doubled twice in the space of 75 years), and in its increasing complexity. Globally, at least 60 million children remain without access to primary school education, and an estimated quarter of a billion are lacking in basic reading and writing skills. The International Labor Organization estimates 200 million people are unemployed in 2013. The labor market demand for routine tasks has fallen rapidly over the past 50 years, meaning we can no longer rely on memorizing and reproducing knowledge acquired from a specific curriculum to support us in our careers.

At U.C. Berkeley, we are building our strategy around trying to make some impact on those big needs. Education remains very much a black box in which inputs are turned into outputs in ways that are difficult to quantify or predict consistently. But that’s no excuse for us or indeed for anyone else involved in education. We need to learn more. At Berkeley, our mission is to help people make progress in their lives through learning. So we had better be sure that we can demonstrate that progress and measure our impact in a meaningful way. Our commitment is to make improving learning outcomes the central driving force of the university, to report on progress and to use the findings to propel continuous innovation and improvement.

We borrowed the term ‘efficacy’ from healthcare. Just as a pharmacist would talk about relieving a tickly cough rather than listing the ingredients of the syrup, we want to be able to state the outcome we help to produce rather than describe the input we provide. Efficacy might be complex enough in healthcare; in education it is frequently many more times so. The process of learning is a social, dynamic, and interactive one. Context — culture, community, language — is central to the learning outcome. We may never be able to ‘prescribe’ an educational process to deliver a desired learning outcome with the same precision as a doctor or pharmacist. Yet all of us involved in education must take a lesson from that most famous phrase in school reports: we must do better.

As we transform our university, our aim is to ensure that our actions, our decisions, and our investments are driven by a clear sense of how we can make a measurable difference on learning outcomes. That idea will shape the choices that we make—what we do, what we stop doing, where we invest and innovate, who we partner with, how we engage with our students, and how we recruit, develop and reward our people.

To get started, we needed a definition. Keeping it simple, we chose to define as follows: a course has efficacy if it has “a measurable impact on improving people’s lives through learning.”

Note that, with this definition, it is the learning outcome that we are pursuing. Passing a test or an exam is good, but it is not an end in itself: what we really want to see is the benefit of doing so in somebody’s life. To give an example, if a student passes an ESL course, that is good. What really matters, though is that their mastery of English helps them to make progress in their career. Or, as another example, achieving an “A” in a class is good, but what really matters is that, as a result, the student can progress to their degree or career of their choice, prepared for work, college, and citizenship.

Central to this effort, we designed our Efficacy Rubric which, instead of imposing a model, asked a set of questions which, once answered by the faculty, would set the course on a path to efficacy. The rubric is in four sections. The idea is that, for each of the four sections, an judgment is reached on a four-point scale. Those four judgments can be combined into a single overall judgement on the likely efficacy of the course. We should emphasize at the outset that the rubric does not claim to be scientific: its basis is in informed human judgment. Crucially, though, it is provocative and challenging. It asks difficult questions; it demands that people think deeply about what they do and why.

Each of the four sections of the rubric is intended to get at a specific vital angle of the course’s efficacy. The brief summary is as follows:

  • Section 1, The Efficacy Goals: What are you trying to do?
  • Section 2, The Evidence: Why should we believe you?
  • Section 3, The Plan: How do you intend to achieve your efficacy goals?
  • Section 4, Capacity: Do you, and those you depend on, have the knowledge, skills, and relationships to deliver the efficacy goals?

The idea is that if the faculty can answer the questions convincingly, then their courses are likely to either already be demonstrating efficacy, or at least to be on the path to efficacy.

This would be a pretty provocative pronouncement for the president of a public university (or any university) to make. It would raise all sorts of questions. At the highest level we might ask, Is efficacy the right model for expressing our educational responsibility to our students? What kinds of conversations would adopting this model lead to among faculty, and how would it influence the way courses are designed and taught? What trade-offs would we make by adopting this particular framework, and do we do more good than harm, on balance, by picking one framework for everyone? Then there would be practical questions. How would faculty begin to apply such a framework as a group, across all departments and courses? Would they even accept the idea that they should? Do they have the skills to do so? What support might they need?

The words above come not from the president of U.C. Berkeley but from Pearson’s report, with some elisions and modifications on my part to make them fit with the thought experiment. I do think that we may ask different questions and, in some cases, answer the same questions differently when we think about the framework in terms of its appropriateness and usefulness for education separately from its context as vendor public relations materials.

Transforming (For? Because Of? Despite?) You

Of course, Pearson is not a public university, and their decision to pursue this strategy as what has historically been a textbook company also raises some different questions. As you think about Pearson declaring that they are now focused on evaluating all their products based on efficacy, one reaction that you may be having is something along the lines of, “Wait. You mean to tell me that, for all of those educational products you’ve been selling for all these years, your product teams are only now thinking about efficacy for the first time?” Another reaction might be, “Wait. You mean to tell me that you think that you, a textbook company, should be defining the learning outcomes and determining the effectiveness of a course rather than the faculty who teach the course?” These questions represent the rock and the hard place between which Pearson finds itself. I will address the former now and the latter in the last section of this post.

It’s impossible to unpack the meaning of Pearson’s move without putting it in the context of the historical relationship between the textbook industry and the teachers who adopt their products. Despite all of the complaints about how bad textbooks are and how clueless these companies are, the relationship between textbook publishers and faculty is unusually intimate. To begin with, I can’t think of any other kind of company that hires literally thousands of sales representatives whose job it is to go visit individual faculty, show them the company’s products, answer questions, and bring feedback on the products back to the company. And speaking of those products, the overwhelming majority of them are written by faculty—many with input from an advisory committee of faculty and pre-publication reviews by other faculty. You can fairly accuse the textbook publishers of many different faults and sins, but not taking faculty input seriously isn’t one of them. Historically, they have relied heavily on that faculty input to shape the pedagogical features on the textbooks. And they have had to, because most of the editors are not teachers themselves. More often than not, they started off as textbook sales reps. If they taught at all, it was typically ten or twenty years ago, and just for a few a few years—long enough for them to figure out that teaching and the academic life weren’t for them. This doesn’t mean that they don’t care about pedagogy or don’t know anything about it, but it does mean that most of what they know comes from talking with their authors and customers.

And by “customers,” I mean faculty, despite the fact that it is the students who actually buy the product. Pearson’s choice to build their learning outcomes effort around a term that comes from the pharmaceutical industry is an historically apt one for the textbook industry. In higher education in the United States, faculty prescribe textbooks and students purchase them. As a result, textbook publishers have generally designed their products to please faculty rather than students. One consequence of this is that they had no need to distinguish product features that offer faculty convenience from those that actually impact student learning. When faculty/customers said to the textbook publishers, “I want my book to come with slides, lecture notes, and a self-grading homework platform so that I don’t have to put as much work into that annoying survey course the department head is making me teach,” then that’s what they provided. Whether that collection of materials had positive impact, negative impact, or no impact on student outcomes was not a question that the textbook publisher had any particular reason to ask. For the most part, the publishers relied on their authors and customers to make good decisions for the students. As long as the they provided the raw materials that the faculty said they needed, the companies’ work was done.

That relationship has been slowly breaking down over the past couple of decades and has now reached a crisis point, for a variety of reasons. The first and most obvious is that students are no longer meekly taking—or at least, and more importantly for the textbook publishers, buying—the “meds” prescribed by their teachers. The used textbook market has been growing since at least 1980s when I was a student, meaning that the number of copies sold of each textbook edition has been going down. For many years, the textbook companies compensated for their losses by artificially creating reasons for new, incompatible editions every few years and by raising prices every year to make up in dollars per book sold what they were losing in number of books sold. This strategy worked for a surprisingly long time, but the publishers have finally hit the price ceiling in the market—aided somewhat by new ways for students to find used books, increasing student willingness to skip buying the books altogether, and increasing faculty willingness to include free or low-cost alternative curricular materials. The textbook industry is in trouble and no longer has the luxury of just trying to squeeze another year out of the old model. Revenues and profits are down. Even Pearson, which is in the best shape of the cohort, have had bad financial results and expect the problems to continue. Most of the major publishers have gone through very significant reorgs, downsizings, and in some cases recapitalization exercises in the past few years. Cengage declared bankruptcy. McGraw Hill sold off their education business to private equity. And so on.

By and large, the textbook publishers have adopted three new strategies to deal with their existential crisis. First, they are diversifying into services. They see that a lot of schools are having trouble, for example, launching online learning programs, so they provide outsourced services to those schools. Second, they are bundling homework platforms and curricular content into one digital platform. If students have to get graded in the homework platform in order to pass the class, then they have to buy the product. And if those homework platforms rely on digital access keys that expire at the end of the semester, then students can’t resell them into the used textbook market. And third, they are finally giving some thought to how they can design products that students actually want to use and buy. They are doing this mostly by trying to learn lessons from other industries about how to design products that make their customers happy. Cengage, for example, has recently undergone a major restructuring to emphasize product management teams in the style of software companies rather than traditional editorial teams.

There is a fourth strategy that is only beginning to emerge. Textbook publishers have noticed that products that have the word “analytics” or “adaptive learning” associated with them sell well. There is still a lot of whiz-bangery to the industry’s thinking, but it is starting to dawn on them that their products might sell better if they can prove that those products actually…you know…work. If they do work. If they provably help people to learn.

Pearson’s efficacy strategy is all about the lessons that they are learning from the success of their MyLabs products. They are trying to change the buying criteria for curricular materials. What if, instead of choosing a textbook based on the reputation of author who wrote it, faculty were to choose the materials they prescribe to their students based on the evidence that those materials actually impact learning? What if students thought of those materials not just as that $150, 10-pound piece of garbage that their professor is making them buy and of which they will only read 10%, but as something that they have reason to believe will help them? What if university administrations and state governments began demanding evidence that students are getting value from the education that they are paying for? (Oh wait; that’s already happening.)

Pearson believes that this world is coming, they want to accelerate its arrival, and they believe that they can position themselves as the market leader in it. As I will outline in the next section, we have plenty of evidence to suggest that they are very, very serious about it. Whether they can actually pull it off is a vastly harder question.

Transforming P(earson)

Many academics who have never worked in a large corporation harbor a stereotype that whatever the CEO says goes (in stark contrast to the academic environment). And there are some companies for which that is mostly true. When I worked at Oracle, I was amazed at how, even in an organization of 120,000 people, when Larry Ellison wanted something to happen, it happened. Even more amazingly, if there was something that somebody in that company wanted to happen, and that somebody wasn’t Larry Ellison or one of his lieutenants, it rarely happened. But in my experience, Oracle is the exception that proves the rule. Most companies have their fiefdoms, their internecine squabbles, their cliques and their cultural inertia that a CEO cannot simply overrule. Let’s start with the sales force. In many companies that have a sales force (including textbook companies), sales reps get either most or all of their compensation not from salary but from commission. Which means that, in an important sense, they don’t really work for the company. They are like franchisees. They are bound by certain rules of their agreement with the company that provides the products that they sell, but they are fundamentally in business for themselves. In such a scenario, which products do salespeople sell? There are two answers to this question: (1) whichever ones they can and (2) whichever ones earn them the biggest commission. Suppose you’re the CEO of a textbook company and you want your sales force to emphasize a new product line that you think has a bright future or that takes your company in a new direction. But your sales rep finds it easier to sell the old product. Which product is she going to sell? Why, the one that’s easier. You can tip the scales a little by providing a bigger commission for the new product, but at the end of the day, your rep is going to weigh that commission against how hard the sale is and how many sales of each kind of product she thinks she can make. Even if you, as CEO, are willing to let the company’s revenues take a short-term hit by emphasizing the product that has better long-term prospects and losing some sales in the process, you can’t force your rep to go along with that decision because she doesn’t work for you. Not really.

So most typical companies that have substantial commission-driven sales forces have at least some significant limits on the CEO’s ability to turn the ship, so to speak. But in this regard, textbook companies are not typical companies. They are much, much worse. All the big textbook publishers got big because they are what business folks call “rollups“. Some physics and engineering textbook publisher somewhere had a really great couple of years and decided to buy a math textbook publisher. That company grew and eventually bought a biology publisher. And so on. But the main business purpose of these mergers and acquisitions was to save costs. Why run two distribution warehouses when you can run one? Why have two printing operations, or two marketing or HR departments? In terms of the core work of producing content, the physics editors have no more reason to talk to the biology editors now that they are under one roof than they did when they were separate. Each small publisher still ran mostly on its own. And that strategy worked just fine until publishers had to do start doing big, whole-company projects like building digital platforms. Then it became a nightmare. But because each individual business unit has its own strong cultures, processes, and internal loyalties, getting them all to line up behind one strategy and set of priorities—particularly in cases where the leader of one business unit had to sacrifice in the interest of helping another—was incredibly hard. Again, in a sense each business unit was a franchise, compensated through its own success and working in its own ways. The history of large textbook companies in the past decade is a history of new leadership trying to centralize power, failing, and then being replaced by new leadership that starts all over again.

So if you’re the CEO of major textbook publisher and you want to unite the entire 45,000-employee company around a plan to transform the way the company does business, what do you do? Surprisingly, Pearson’s CEO John Fallon’s answer was, “I’ll create a rubric.”

I’m not going to analyze Pearson’s rubric in detail here because (a) this post is already ridiculously long even without that analysis, and (b) there are plenty of others in the educational community who are incredibly well qualified to offer a critique, and I’m hoping that somebody will. (Pearson has an online tool for applying the rubric if you want to give it whirl.) I’ll say this much about it: It’s nothing special. It’s not bad, but it’s not genius either. There are plenty of flaws and limitations you could find if you worked at it and applied it broadly enough. There is no magic in it.

But here’s the thing: There is never any magic in a rubric. The magic, when there is any, happens from the norming conversations that the rubric engenders. It happens when one colleague says to another, “What do you mean by ‘quality of evidence’?” Or “I scored that course a 2 on effectiveness. Why did you think it was a 4?” To the degree that the Effectiveness Framework proves to have any magic for Pearson, it will be in the norming conversations that it engenders across the company. Like our hypothetical Berkeley president, Fallon is working with diverse groups within an institution that has a culture of independence and Balkanization. Some of this is for good reason; conversations about effectiveness in chemistry education should look very different from conversations about effectiveness in fine arts education. Some of the fractiousness is about lack of a common culture and language necessary to discuss what otherwise are common challenges. And some of it is just human territoriality and self-interest. The first two challenges might be addressed by having a deep and wide ongoing norming conversation about a rubric that is general enough to cover a wide range disciplines and products but focused enough to provoke important discussions. The goal is for that conversation to become the basis for a new culture. The third challenge might be addressed by reinforcing that culture through your HR and other business practices.

The actions spelled out in Pearson’s document demonstrate that the company is making a substantial investment in activities that should be conversation- and culture-building:

  • They recruited volunteer business leaders within the company who were interested in piloting the framework.
  • The created a process by which the product teams under those business leaders conducted self-assessments of their products using the framework. Typically six to eight members of each team participated in the reviews.
  • Each product team would norm their self-evaluations with the central efficacy review team.
  • After the first few reviews, Pearson began running training sessions on the review process for volunteers—in fact, prospective attendees had to apply—in at least five locations around the globe.
  • To date, the company has conducted 100 efficacy reviews across 15 countries and trained 600 reviewers across 25 countries.
  • They have developed the capacity to conduct 150 new efficacy reviews per year.
  • They have recruited a volunteer Efficacy Steering Committee of people who “lead efficacy for entire regions or business units on top of their day jobs.”
  • They have developed a workshop to practice applying the efficacy framework which they have delivered to 5,500 participants.
  • They are developing an e-learning module for training on the framework.

It’s hard to argue that internal activity of this scope is just PR. This is internal-focused effort, not just decorations for press releases. How much of it is genuine culture-building? It’s hard to know from outside, but the company took pains in the document to talk about the importance of fostering “debate” and building enthusiasm that inspires employees to participate rather than forcing them.

In addition to the actions themselves, the document gives us some clues as to how Pearson executives characterize what all these efforts add up to:

To convene a company around a single idea, you need to do three essential things: realign the existing portfolio around the idea, set the routines in place to embed that idea in the future, and build a community to make the change irreversible.

Reviewing the existing portfolio will help steer the current business towards delivery better outcomes. But in order to have perpetual change, we also need to shift an idea from the periphery to the centre of the organisation. You need to ensure that it’s a recurring thought, day in, day out, and a part of every employee’s routine.

In fact, you need to deconstruct the essential routines that form the core business processes of a company, and reconstruct them around the idea you’re seeking to embed. It’s the only way to ensure that future generations of employees and leaders across the company hold the values on which you depend.

We will need to address each of these to deliver ‘institutionalisation’ of efficacy at Pearson, taking all the elements of the company that keep the core business engine running and ask ourselves: “What if we could redesign these processes from scratch to ensure they resulted in delivering learning outcomes?”

If we take them at their word, they are trying to rebuild the company from the ground up. Again, the scope of the effort is consistent with the rhetoric. In addition to the cultural work, they are making significant changes to their business processes:

  • They have created a global educational research function in the company, as well as a research center for each of three major areas—Schools, Higher Education, and Professional Learning.
  • They are developing a repository of internal and external educational research that will be available to “business leaders” (presumably this means within Pearson).
  • They have developed a partnership with Nesta to define standards of evidence in educational efficacy research.
  • They have built a formal efficacy evaluation into their business acquisition process which applies to every acquisition above $3 million.
  • They apply an efficacy review to internal product investments that are over $1 million and to selected smaller projects that have strategic importance.
  • They have developed efficacy review tools that have to be applied on both bids and post-delivery reviews for bids on large-scale projects.
  • They have added statements about commitment to efficacy in their job advertisements, made efficacy a topic in interviews of prospective employees, and added efficacy training to employee orientation.
  • They are adding efficacy criteria to employee performance reviews and, in some cases, linking efficacy results to compensation.
  • They are including efficacy updates in their monthly progress reports to their board of directors.

So we have plenty of reason to believe that Pearson is quite serious about becoming a company whose mission is to deliver educational efficacy, whatever that may mean to them. But that leads us to several more questions. First, does Pearson’s notion of efficacy truly align with the academic community’s ideas of what a good education is supposed to accomplish? And second, will Pearson be successful as a company if they deliver “efficacious” educational products? To my mind, these are the same question. Pearson can succeed in the market with this strategy to the degree that they build products which accomplish educational goals that educational stakeholders agree is important, as proven by measures that educational stakeholders agree are valid and significant.

And this brings us to a huge gap in Pearson’s thinking about efficacy to-date.

The Other Half of the Job

Let’s think some more about the analogy to efficacy in health care. Suppose Pfizer declared that they were going to define the standards by which efficacy in medicine would be measured. They would conduct internal research, cross-reference it with external research, come up with a rating system for the research, and define what it means for medicines to be effective. They would then apply those standards to their own medicines. And, after all is said and done, they would share their system with physicians and university researchers in the hopes that the medical community might be reassured about the quality of Pfizer’s products and maybe even contribute some ideas to the framework around the edges. How confident would we be that what Pfizer delivers would consistently be in the objective best interest of improving health? This is not entirely hypothetical; much of the drug research that happens today is sponsored by drug companies. Unsurprisingly, this state of affairs is viewed by many as deeply problematic, to say the least. It certainly doesn’t help the brand value of Pfizer. But at least much of that medical research is conducted by physicians and academic researchers and is subject to the scientific peer review process. Pearson is creating their framework largely on their own, selectively inviting in external participants here and there.

I get why they had to do this. The company is bleeding money, it will take some time to stop the flow of blood, and they couldn’t wait to build consensus before they tied the tourniquet. But it is not going to get them where they want to go. While there are obvious concerns about ethics and about whether such a company-driven approach is fundamentally compatible with progress on complex questions such as defining what an education is good for and how we know when we have achieved these ends, I want to focus on the business aspects of the problem. I want to focus on why continuing down this path is bad for Pearson. Or rather, why driving hard toward becoming facilitators rather than owners of efficacy research is good for Pearson.

I could give a number of examples, but one should hopefully suffice. In preparing to write this post, I asked Annie Cellini, Pearson’s Senior Vice President of Marketing and Strategy, whether Pearson intends to share the completed rubrics for their products with customers and prospects. This was her reply:

Though we don’t plan to share product efficacy scoring as part of our sales and marketing materials per se, where a product has a strong research and evidence base, we will communicate that to customers. It’s also worth saying that the most important output of an efficacy review isn’t a rubric score. We believe that much of a review’s value comes from the conversations that it prompts teams to have, which focus on the path forward, and on how to improve the product or service from a learner perspective. A poor score does not mean the product doesn’t work well. It often means that teams are not collecting the type of data needed in order to get a sufficiently robust view of the product’s efficacy, or they may not have a sufficiently practical plan to continuously enhance the product based on data. Their improvement plan will encourage them to start gathering new information, to start working in new ways, and to make sure that their customers are aligned with the outcomes they plan to achieve and understand their role in the product’s path to efficacy.

This is a perfectly sensible and responsible reply if you believe that the main value of the Efficacy Framework to customers is in the data that results from the work a product team does after an efficacy review. But remember, the magic of the rubric is in the norming conversations. Annie’s reply suggests that Pearson understands this in terms of the Pearson-internal processes but not yet in terms of their relationships with their customers. If Pearson were to say to faculty, “Here’s what we think we know about the efficacy of this product, here’s what we don’t know yet, and here is how we are thinking about the question,” they might get a number of responses. Maybe they would get, “Oh, well here’s how I know that it’s effective with my class.” Or “The reason that you don’t have a good answer on effectiveness yet is that your rubric doesn’t provide a way to capture the educational value that your product delivers for my students.” Or “I don’t use this product because it has direct educational effectiveness. It frees me up from some grunt work so that I can conduct activities with the class that have educational impact.” Most of all, if you’re John Fallon, you really want faculty to say to their sales reps, “Huh. I never thought about the product in quite those terms, and it makes me think a little differently about how I might use it going forward. What can you tell me about the effectiveness of this other product that I’m thinking about using, at least as Pearson sees it?” And you really want your sales reps to run back to the product teams, hair on fire, saying “Quick! Tell me everything you know about the effectiveness of this product!”

Pearson won’t get that conversation by just publishing end results of their internal analysis when they have them, which means that they have a high risk of failing to align their products with the needs and desires of their market if they think about the relationship between their framework and their customers in that way. I don’t think Pearson fully gets that yet. While the authors of The Incomplete Guide frequently invoke terms like “community” and “leaders” in the document, they generally seem to mean the community and leaders within Pearson. The company’s efforts to reach out to the academic community for feedback and participation are generally framed as an extension of their efforts rather than the very heart of them. And yet, Pearson’s brightest possible future is not as a company that designs educationally effective products, but as one that facilitates conversation and research about efficacy within the broader academic community (and in so doing is able to design products that their customers agree are effective for important educational goals as determined by meaningful measures).

There are a number of reasons why this part of the transformation will be at least as difficult as the part that Pearson is undertaking now. First, it is far from clear that the company has the trust of the academic community that would be necessary for them to take such a role. That would have to be built, in some cases from the ground (or even the basement) up. Pearson does have real strengths that are known within certain segments of the academic community—in data science, for example—but this does not transfer to a general reputation. Second (and relatedly), unlike the medical research community, the educational research community is still nascent and fragmented. Finding non-paternalistic but effective ways to bring that community together and facilitate useful conversations will be difficult to say the least. These two challenges are outside the company’s sphere of control, which means that Pearson will have to develop new ways to think about how to build their relationships with the broader educational community. Mike Caulfield wrote a post a while back about Eric von Hippel’s work on customer innovation, which is one potential font of inspiration for the company. But make no mistake; this will be a tough nut to crack.

Internally, changing the way they think about answering the questions that the framework asks them will entail as much subtle, difficult, and pervasive re-engineering of the corporate reflexes and business processes as the work being undertaken now. As I described earlier, all textbook companies that have been around for a while are wired for a particular relationship with faculty that is at the heart of how they design, produce, and sell their products. Their editors have gone through decades of tuning the way they think and work to this process, and so have their customers. When Pearson layers a discussion of efficacy onto these business processes, a tension is created between the old and new ways of doing things. Suddenly, authors and customers don’t necessarily get what they want from their products just because they asked for them. There are potentially conflicting criteria. The framework itself provides nothing to help resolve this tension. At best, it potentially scaffolds a norming conversation. But a product management methodology that can combine knowledge about efficacy, user desires, and usability requires more tools than that. And that problem is even worse in some ways now that product teams have multiple specialized roles. The editor, author, adopting teacher, instructional designer, cognitive science researcher, psychometrician, data scientist, and UX engineer may all work together to develop a unified vision for a product, but more often than not they are like the blind man and the elephant. Agreeing in principle on what attributes an effective product might have is not at all the same as being able to design a product to be effective, where “effective” is shared notion between the company and the customers.

Pearson will need to create a new methodology and weave it into the fabric of the company. There are a number of sources from which they can draw. The Incomplete Guide mentions Lean Startup techniques, which are as good a place to start as any. But there is no methodology I know of that will work off-the-rack for education, and there certainly is no talent pool that has been trained in any such methodology. I have worked with multiple educational technology product teams in multiple companies on just this problem, and it is very, very hard. In fact, it may be the single hardest problem that the educational technology industry faces today, as well as one of the harder problems that the larger educational community faces.

We should not underestimate the scope of the effort that Pearson is making, but neither should they underestimate the scope of the challenge that is yet to come.

I have often said that corporations are amoral in the same sense that lawnmowers are amoral. They do what they are designed to do. Pearson has published a remarkably detailed blueprint of how they intend to rebuild their machine from the ground up. And in doing so, they have revealed just how hard a task they have in front of them.

Share Button

Google+ Comments

About Michael Feldstein

Michael Feldstein is co-Publisher of e-Literate, co-Producer of e-Literate TV, and Partner in MindWires Consulting. For more information, see his profile page.
This entry was posted in Ed Tech and tagged , , , , , , , . Bookmark the permalink.

26 Responses to Can Pearson Solve the Rubric’s Cube?

  1. mellgrim says:

    Thank you so much for this post. The analysis is great and the further resources are a great read. I went in to play with their rubric and found that well, it is teaching (pun intended) to suck eggs. It’s asking me as a teacher to assess myself in the same way I do to align with a range of requirements including the Teaching Quality Framework ( and the Australian Professional Standards for Teachers ( And more to the point teachers are always focused on the teaching output. There is no conversation flipping to be had. It’s what we do. If Pearson are only catching on to the concept now, then perhaps they shouldn’t be in the education game. My other concern is the positioning of Pearson to become the only game in town. The Internet provides teachers with resources and Michael points out, it feels like they want to minimise this option for teachers by creating path dependancy. I’m new to the higher education concept but in high schools, there can be too heavy a price paid for relying on one source. Anyway, two cents worth when all I wanted to day was thank you!

  2. dkernohan says:

    “Efficacy” is deliverology. To make sense of this move by Pearson you need to look at Michael Barber’s deliverology project in UK government. It’s a straight reapplication of the same principles… by the same person.

    I agree it is an interesting move by a publisher (which has major aspirations to become a soup-to-nuts education provider) but in the context of Barber’s career it is only shocking in that it didn’t happen sooner.

  3. mellgrim and David, you make a couple of good points here. Regarding the focus on outputs vs. inputs (or something else), the situation in K12 is substantially different than higher ed (and different from country to country as well). Here in the United States, it’s fairly easy to make the case that we have swung (way) too far toward an obsession on easily measurable (and often insignificant) outputs in K12, while we paid not nearly enough attention to any kind of outputs at all at the higher education level. It’s also quite different for a company to declare that the products that the designs of their products are going to be heavily driven by their ability to produce a measurable impact on learning than it is for schools and governments to declare that they are going to judge education reductively by those standards. Context is everything.

    One problem that we have is that the dividing line between these contexts is porous, which brings us to your second point, mellgrim. All companies try to make their products essential. That’s what they are built to do. The very best companies do this by building products that are so great people just want them. But most companies, including textbook companies, are not very good at that, so they resort to other means, some of which are less savory than others. It will be a natural instinct of any big textbook company to try to sell their framework as the right framework for education in general, not making the context distinction that I am making here. It doesn’t even matter how careful the executives are not to make this elision; the sales reps will make it on their own. That sets up a fundamental tension between the educators and the vendors. Much of this is unavoidable under a capitalist system; as long as there are vendors, there will need to be consumerism (in the positive sense of the word). One of the reasons that I would like to see Pearson invite educators into their internal conversations as quickly and deeply as possible is that it shifts some of this inevitable tension to the norming conversations, where it can be productive, and out of the sales pitch, where it rarely is.

    With respect to Deliverology, yes there are some unsurprising similarities in approach. (I did some reading after watching your video, David.) But again, context is everything here. I don’t have any problems with asking the questions that the Effectiveness Framework asks. To the contrary, I think they are good questions to ask. Whether asking them does net good or harm depends on how you go about answering them and what you do with those answers. To my mind, the danger is not in the approach itself but in the reductive application of it.

  4. allison390 says:

    Another great post, Michael. Thank you for taking the time to address this so thoughtfully. We find parallels (on a much smaller scale, of course) with the way we invite customers into our own PM methodology. Look forward to following Pearson’s progress on this – and your take on their relative success, of course!

  5. Laraine says:

    For starters, thank you so much for this terrific post, Michael, which made me pay attention to Pearson’s emphasis on “efficacy,” an emphasis I didn’t think especially important until you wrote about it in such detail.

    If I read Pearson’s definition of “efficacy” and their mission statement correctly, then some of the educational goals I care about–intellectual self confidence that allows for mistakes and learning from them; inquisitiveness about and openness to other cultures; a healthy skepticism concerning all traditional wisdom; the willingness to question, even challenge both authority and group consensus–won’t get much play in their educational model.

    These intellectual goals aren’t especially measureable, or at least not with the precision that Pearson’s model seems to demand: “Do you have ambitious and MEASUREABLE TARGETS in place, and DEADLINES for achieving them?”Rated according to the Pearson framework, a lot of my “anecdotal evidence” about say student willingness to question authority would get a “red” rating, which despite this not being stated explicitly, seems to mean “poor.” (As an aside, the whole color coding of sample responses in the Efficacy Framework Tool reminds me way too much of the war on terrorism “alerts.”)

    If this were any other company except Pearson, I wouldn’t get my knickers into such a knot over their efficacy rubric with its lead-footed emphasis on measureable outcomes achieved in pre-determined time frames (and tellingly without any “evidence used … from a source considered to be highly controversial”) or the grandiose tone of their Path-to-Efficacy site:

    “It will be a matter of how the decision-makers in the global education community act collectively to shape the evolving direction of the education debate. PEARSON [caps are mine although they fit the tone of this manifesto], as the world’s largest company, has both the responsibility and the potential to pursue this conversation.”

    But given that this is the most powerful publisher in the education market, I’m kind of worried about how much Pearson’s vision of education will influence the schools, texts, and technology tools of the future. Although your post doesn’t say so explicitly, it does suggest, to me anyway, but my thinking might be wishful, that the company is over-reaching. I hope that’s true, and I hope that, as a result, they end up backing off a good deal from a vision of education that personally gives me the heebie jeebies, even if it is lifted, as someone else says above, from Michael Barber, whom I never heard of but now will definitely look up.

  6. Pingback: New Year’s Surprises | Joyful Latin Learning -- Tres Columnae

  7. Michael Evans says:

    The implication that a stated response to a rubric, even including the normative discussions that ensue, is a “desired state” is problematic in all endeavours. Rubrics may be most effective in their ability to assess directions for continual improvement, not a terminal description. This would be an advisable approach in that it also continually allows for an evolution of methodology, e.g, the eventual partnering of business and education in identifying mutually changing definitions of efficacy.

  8. Pingback: bookmarks for January 1st, 2014 through January 2nd, 2014 | Morgan's Log

  9. Shawn Rubin says:

    Thank you so much for this article. As someone who works closely in the k-12 space with Pearson reps and has been hearing about their new “Efficacy” approach with much confusion and little interest it was awesome to read your breakdown.

    I find this interesting through a different lens. As a k-12 person I see more and more reliance on rubrics like Pearson’s efficacy rubric across education reform efforts. Whether it’s the teacher evaluation model, the rating of edtech products against common core, or even looking at student work during the RTI process, everyone touts the value of the conversation that happens during the rubric discussion as the end all be all.

    In this article Michael lays out the time, energy and I would imagine insane amount of dollars that Pearson has had to invest to create an effective norming process using rubrics and “conversation”. What district, school or PLC in the K12 space has the resources or people capable of tackling this with the gusto that is described above? If this what it takes to be effective using a rubric process are we just kidding ourselves in K-12?

  10. Michael, the point you make is similar to one made by Carrie Saarinen in her (excellent) blog post in reply to this one.

    Shawn, while I don’t think you need all Pearson’s resources to do rubric norming, you make a good point that it does take a significant investment of time and organizational effort. I would not be surprised if schools are unable to make that investment a fair percentage of the time.

  11. Pingback: Efficacy and the Big Picture | Cultivating Minds

  12. Jack Gunther says:

    Thank you for this excellent assessment of Pearson’s Efficacy strategy. Was there any discussion with them about a possible Section 5: How will we know if your Efficacy goals are achieved?

  13. Thanks, Jack. This post isn’t based on conversation with Pearson; just reading their documentation (and attendance of one public event, although that mostly provided background for the post rather than substance). But you’re pointing to a critical section of the document, and one that I will likely blog about at some point.

  14. Dr. Brown says:

    Thank you Michael. My colleagues and I have talked about this for a while – we were alerted to the new “efficacy’ framework by a few different Pearson representatives (note – not book reps, but data scientists, conference presenters, etc) over the past few months.

    Per your “case study” reference, Pearson seems to be attempting an IBM-type makeover. And on the surface, I appreciate the rhetoric and the notion. I believe they WANT to make this happen. (I’ll also say that I’m not a vendor-hater. I am amazed at how many of my fellow academics loathe all makes of education materials simply because they exist! So, taking a pragmatic approach to education – and life – I really try to look at each vendor and each product in context.)

    So, when I say that I don’t think they are going to make it work, it’s not because I don’t want it to work. I hope for the sake of education around the globe it does. I’m not offended that they want to put a stake in the ground around efficacy, rubrics, outcomes, or any other education concept. It’s not really rocket science, and there is almost zero evidence to show that professors or school teachers are the only ones who can make this happen. (Ironically, much of the public believes academics are the problem, not the solution and that companies like Pearson SHOULD be doing this…but that is another debate.)

    But my problem is simply two clues. First, if you look at what IBM did to reinvent themselves, they sold the hardware business. And to your point here, they really had to. It’s like Cortes burning the ships when he arrived in the new world. Had he not done so, his men would have always looked back. Had IBM not sold that business, the sales force would have always looked back and likely fallen back into old habits and old patterns. So, why then hasn’t John Fallon sold the book business? I know they have blown up the Pearson of a few years ago (including the Financial Times and Penguin Books), but why didn’t they sell off the book business? It’s a tired, old albatross that is only going to tempt old patterns and bad habits. And to boot, there are plenty of other options the company could begin to use! They have a surprisingly strong research organization (think empirical research by actual academic scientists, not R&D). As well, Pearson seems to have a robust digital business looking at adaptivity, personalization, and on and on. How hard would it be to actually build on those strong businesses to provide significant gaming options for educators or to up the ante on the MOOC market, perhaps cracking the nut in terms of large scale education? But instead, they maintain their tests, their books, and their reps…which brings me to my second clue.

    I know some people at Pearson. I have talked to some scientists and data visualization experts from Pearson recently. In fact, I was blown away (as were 500 other people) at an eLearning conference by a keynote presenter who was actually a Pearson executive – name withheld as I don’t want to get him into any trouble here. But in all my dealings with Pearson people and in doing some research, it seems that John Fallon has transformed the organization from people who know books, publishing, and delivery into one that is focused on efficacy, service, and research as run by people who know books, publishing, and delivery!!! Why didn’t Pearson steal executives from Zappos, Amazon, IBM, Oracle, Pfizer, or any other group who could provide actual leadership in the areas they claim are important? Why haven’t they promoted some of the genius rock stars inside the company who have new ideas and fresh views, replacing the dinosaurs that got them where they are today? They exist. I’ve talked to them.

    I’ll give them Michael Barber. That was a healthy steal and he has obviously been the puppet master behind much of this transformation. But I’m sorry, I don’t buy that one guy – not even the CEO – can make this kind of systemic change. Again, I’d like to see it happen. TRULY! But until some new leadership takes the helm, sells the tired, old product lines, and makes this more than a new system to game by employees, this is all just going to be a costly mess.

    I hope we will see more about this in the months ahead. It’s a great conversation for the academy to have, regardless of who initiates it. Thanks for the solid analysis.

  15. You raise some good questions, Dr. Brown, but I think the answers are complicated. Regarding the IBM analogy, let me ask you this question: Who would buy Pearson’s analog textbook business? In IBM’s case, while hardware was commoditizing, nobody was arguing that it was going to go away altogether. Cengage and McGraw Hill might be interested in cherry picking a handful of the most popular titles, but that would fall far short of divestiture for Pearson. I don’t know the numbers specifically for that part of their business, but I suppose it is possible in principle that some PE firm might be interested in milking it for the cash until it dies. I doubt it, but let’s entertain the notion for the sake of argument. Paradoxically, while that business isn’t worth much to anyone else, it is also worth too much to Pearson. We are at a weird point in the digital transition in higher education. While most of the profit is in the sale of digital products, they often still attach to sales of analog books. And even when the digital transformation is complete, the publishers will still need high-quality content. I suppose they could release their content as OER. That would be an interesting and provocative strategy. But it’s a vastly riskier strategy than IBM’s selling off of the hardware business, at a time when Pearson is already taking some big risks (and losing money).

    Regarding talent, Pearson has been making new hires at the VP level and above that are increasingly people with non-traditional backgrounds (in an industry that overwhelmingly hires executive talent from their close competitors). I’m thinking of people like Curtiss Barnes, Lisa Lewin, Aref Matin, and Saad Rizvi. But if you look closely both at the biographies of these people and at the positions that they are in, it’s pretty easy to see that this hiring is being done very carefully to balance new perspectives against having at least some background in education. Having worked at Oracle for five years, I can only think of a couple of executives there who I think would actually be additive to Pearson (and Pearson has already hired one of them). Education is complicated, both in terms of its business characteristics as an industry and in terms of how one builds good products for it. Josh Coates has done an exceptional job coming in from the outside and leading Instructure, but LMS companies are much less complex in terms of their relationships to content and disciplinary particularities than a company like Pearson is. Pearson absolutely needs content people, and they will continue to need them long after their textbook business is dead. Maybe they could be more aggressive about bringing in new blood—I don’t have a firm enough grasp of what they’re doing to say either way with any confidence—but I’m not at all convinced that a wholesale turnover of executive talent with tech giant immigrants would be a net benefit for them.

  16. Dan says:

    Great stuff

  17. Dan says:

    Without a change Person is done in 2 Yrs

  18. Don says:

    I spent 40 years in the college publishing industry. The last 14 years with Pearson. This is the bottom line. No matter how many consultants or VP’s your hire or how many press releases you issue, it all comes down to the best written book by authors who have a passion for teaching. That is the book that will get adopted 90% of the time.
    Don Beville

  19. Pingback: Gen Ed and Competency-Based Education | e-Literatee-Literate

  20. Pingback: Pearson Redefines Its Goal as “Efficacy” | Diane Ravitch's blog

  21. titleonetexasteacher says:

    Pearson is becoming or has become a monopoly. Why isn’t this illegal?

  22. Special Educator says:

    Efficacy. Quite possibly the most vacuous word in the English language. Given Pearson’s history and the clearly opportunistic timing of this “change,” I expect many good educators will simply roll their eyes and keep on teaching. At this point, we’re just patting the heads of those in the poorly run and rarely insightful education measurement world.

  23. Pingback: An Employee View of Textbook Publishers and Ed Tech Companies | e-Literatee-Literate

  24. Pingback: Efficacy, Education, and the Last Mile | e-Literatee-Literate

  25. Pingback: Efficacy Math is Hard | e-Literatee-Literate

  26. Pingback: Efficacy and evidence | Adaptive Learning in ELT

Comments are closed.