By Phil Hill
By reading the SJSU research report (download actual report here), one item that really hits me is that however different the scaling model is for MOOCs, they are still online courses and have similar success factors. I am not trying to minimize the value of the report by the title of this post, because there is real value in letting objective data lead you to conclusions, even if hunches and assumptions would have led to similar results.
First, however, it would be useful to understand the data and key terms – AOLE is Augmented Online Learning Environment, which represents the additional support structures and for-credit status for MOOCs.
Out of a total of 274 students enrolled in AOLE courses, 249 students remained in the sample after data cleaning, which included removal of students enrolled in multiple courses and of those with no course activity. In addition, 36 students were removed who withdrew from the course or received a final grade of Incomplete. This left 213 students for the deeper analysis. This data set will be referenced throughout this report as a “research data file”.
So this data does not include no-shows, withdraws or incompletes, making it more akin to analyzing a traditional college course after the add / drop period.
Table 6 shows the pass rate (getting a grade of C or above) broken down by course and by matriculated / non-matriculated status.
We can see that matriculated students performed much better as a group, with pass rates of 30%, 50% and 54% for each course. Non-matriculated students (45% of whom were high school students from an urban environment) had pass rates of 18%, 12% and 49%.
It is also worth noting that MATH 6L was a remedial math course, MATH 8 was college-level algebra and STAT 95 was elementary statistics. MATH 6L deserves special consideration:
Matriculated students who do not pass Math 6L during their first semester will be allowed to repeat it once. If they do not pass this course by the end of their first year, they will need to complete this course at a community college before they are eligible to enroll at SJSU. However, due to budget restrictions, MATH 6L has only been offered at SJSU in fall semesters since Fall 2009 so students that don’t pass Math 6L in the fall semester do not have the option of retaking it in the fall semester do not have the option of retaking it in the spring semester at SJSU. The Udacity-SJSU 6L course ￼￼offered an alternative for students in this situation. All matriculated students in the course had failed Math 6L before.
The report studies much more than just completion and pass rates, as it looks at student activities and looks for “significance of associations between individual predictor variables and pass/fail”.
The primary conclusion from the model, in terms of importance to passing the course, is that measures of student effort eclipse all other variables examined in the study, including demographic descriptions of the students, course subject matter and student use of support services. Although support services may be important, they are ￼overshadowed in the current models by students’ degree of effort devoted to their courses. This overall finding may indicate that accountable activity by students— problem sets for example—may be a key ingredient of student success in this environment. [snip]
Students who work harder do better in the courses.
Table 12 shows the 5 variables that are highly significantly related to pass/fail for 6 different groupings of the students. Four variables are measures of effort and the other variable measures use of support services. Their functional behaviors relative to pass/fail follows the table.
An interesting finding was that support usage was not a strong predictor of success, but not that the issue was unimportant. Rather than the total amount of support used by students, the actual awareness of support structures was the real issue.
While the regression analysis did not find a positive relationship between use of online support and positive outcomes, this should not be interpreted to mean that online support cannot increase student engagement and success. As students, Udacity service providers and faculty members explained, several factors complicated students’ ability to fully use the support services, including their limited online experience, their lack of awareness that these services were available and the difficulties they experienced interacting with some aspects of the online platform. It is thus the advice of the research team that additional investigations be conducted into the role that online and other support can play in the delivery of AOLE courses once the initial technical and other complications have been addressed.
The big comparison that has some value is how the students within the MOOCs performed compared to historical rates for traditional face-to-face courses. Keep in mind, however, that MATH 6L has not actually been available since 2009, so there is an argument that the real comparison is a MOOC of MATH 6L versus nothing, at least at SJSU. Table 7 summarizes this comparison.
The key measures for matriculated students are MOOC vs. face-to-face pass rates of 30% vs. 34 – 50% for MATH 6L, 50% vs. 52 – 73% for MATH 8 and 54% vs. 71 – 80% for STAT 95. The MOOC pilot led to pass rates below those of the face-to-face courses, but not dramatically lower rates.
So where does this leave us?
There is more information available, and I look forward to reading other analysis posts. As I mentioned earlier, however, what really strikes me is that many of the findings match what we already know about online courses in general.
- Remedial students have the toughest time self-regulating and performing well in an online environment. If these students are targeted for a program, there needs to be significant effort spend on support structures that proactively help students and don’t rely on their passive use (push vs. pull).
- Without significant design and support, online courses tend to have lower pass rates than their counterpart face-to-face courses (although in many online programs with real history, the pass rates can be roughly equivalent).
- Students need to participate in online courses to succeed, just as is true for face-to-face courses.
MOOCs are online courses, and many, if not most, of the known best practices for online education also apply to MOOCs.
I commend the research team and SJSU for providing this report.