Pilots? We don’t need no stinkin’ pilots!

Timothy Harfield commented on Arizona State University’s approach to pilots and scaling innovation at ASU.

excellent comment on the problem of scaling innovation in #HigherEd. This is a central concern for @UIAinnovation.

The University Innovation Alliance is “a consortium of 11 large public research universities committed to making high-quality college degrees accessible to a diverse body of students”. I wrote about this “central concern” last summer in a post titled “Pilots: Too many ed tech innovations stuck in purgatory”, using the frame of Everett Rogers’ Diffusions of Innovations model. While the trigger for that post was on ed tech products, the same situation applies for the course design situation.

5 Stages of Adoption

What we are seeing in ed tech in most cases, I would argue, is that for institutions the new ideas (applications, products, services) are stuck the Persuasion stage. There is knowledge and application amongst some early adopters in small-scale pilots, but majority of faculty members either have no knowledge of the pilot or are not persuaded that the idea is to their advantage, and there is little support or structure to get the organization at large (i.e. the majority of faculty for a traditional institution, or perhaps for central academic technology organization) to make a considered decision. It’s important to note that in many cases, the innovation should not be spread to the majority, either due to being a poor solution or even due to organizational dynamics based on how the innovation is introduced.

The Purgatory of Pilots

This stuck process ends up as an ed tech purgatory – with promises and potential of the heaven of full institutional adoption with meaningful results to follow, but also with the peril of either never getting out of purgatory or outright rejection over time.

Back to Timothy’s comment. He was specifically commenting on Phil Regier’s interview in the e-Literate TV case study on ASU.

Phil Hill: There are plenty of institutions experimenting with new technology-based pedagogical approaches, but pilots often present a challenge to scale with quality. ASU’s vision, however, centers on scale and access. One observation I’ve seen from what’s happening in the US is there are a lot of pilots, but that never scale to go across a school. You sound confident that you will be scaling.

Philip Regier: We kind of don’t pilot stuff here. When we did the math program, we actually turned it on in August 2012 after all of nine months of preparation working with Knewton.We turned it on, and it applied to every seat in every freshman math course at the university. And there’s a reason for that. My experience—not just mine, but the university’s experience with pilots is that they have a very difficult time getting to scale.
Pilots … have a very difficult time getting to scale.

Part of the reason is because, guess what? It doesn’t work the first time. It doesn’t work the first time, maybe not the second. It takes multiple iterations before you understand and are able to succeed.If you start with a pilot and you go a semester or two and it’s, “Hey, this isn’t as good as what we were doing,” you’ll never get to scale.

In our case, the experience with math is a very good example of that because working with a new technology is not a silver bullet. It’s not like we’re going to use this technology, and now all of the grades are going to go up by 15 percent. What you have to do is work with the technology and develop the entire learning ecosystem around it, and that means training faculty.

That’s one approach to the scaling innovation challenge that affects not just the University Innovation Alliance institutions but most schools. This approach also raises some questions. While Phil Regier stated in further comments not in the episode that faculty were fully involved in the decision to implement new programs, are they also fully involved in evaluating whether new programs are working and whether changes are needed? Does this no pilot approach lead to the continuation of programs that have fatal flaws and should be ended rather than changed?

It is, however, an approach that directly addresses the structural barriers to diffusing the innovations. Based on Phil Regier’s comments, this approach also leads to investment in and professional development of faculty members involved.

Share Button
"Pilots? We don't need no stinkin' pilots!", 5 out of 5 based on 5 ratings.

Google+ Comments

About Phil Hill

Phil is a consultant and industry analyst covering the educational technology market primarily for higher education. He has written for e-Literate since Aug 2011. For a more complete biography, view his profile page.
This entry was posted in Ed Tech and tagged , , , , , , , , , . Bookmark the permalink.

5 Responses to Pilots? We don’t need no stinkin’ pilots!

  1. 1) For more about faculty reactions to the math technology introduced at ASU, one can look at the following article in Inside Higher Ed:

    The reactions were more nuanced that a “yes/no” dichotomy.

    2) I am a course coordinator for a large intro math class and our department went all in for an online homework system mandated across all sections for that class. If we ran pilots, we would be in pilot mode forever. We are finding that instructors with more enthusiasm for technology had higher completion rates with the online homework than those who didn’t. So staffing these sections accordingly is one of the issues to consider.

  2. Phil Hill says:


    Thanks for reminder of that IHE article.

    When you say that your “department went all in”, did you introduce in a big bang, or did you roll out over time? Not using pilots (essentially committing up front to a pathway instead of small trial and delayed decision) is different than big bang. Curious to hear a little more on your experience here.

  3. No roll out over time. We required the online component – homework only -for all sections last Fall semester. The online homework system is a well established one, and required only the coordinator to have some level of platform knowledge (for what we wanted to do, anyway). The section instructors could view student progress etc., but I managed all the due dates, assignments etc. It was relatively painless, as the software was easy to use and administer, and we plan to continue. If interested in specific details, I ‘d be glad to share them via email.

  4. Phil Hill says:

    Thanks – useful info. I don’t think I need more details, but was curious on your approach at high level.

  5. Pingback: Universities As Innovators That Have Difficulty Adopting Their Own Changes -e-Literate

Comments are closed.