9 responses

  1. lauragibbs
    August 21, 2013

    Thanks as always for the reporting… and, as always, I find this so totally alien from the data approach that I would take if I could wave my magic wand and turn Desire2Learn into a flexible system for gathering data that I would want to know, and for reporting that data both to me and to the students. The most important reporting, in my opinion anyway, is the data that goes straight back to the students, independent of teacher or institutional mediation. Thank you for talking about that issue here. It troubles me A LOT. And for MOOCs, it’s obvious that we have to be student-centric. So maybe that will be an issue that the MOOCs will affect positively (and heck, we should get at least something good out of all the MOOC frenzy).

    Here’s my question: for the kinds of classes I teach, it’s obvious to me that the key to student success is ENGAGEMENT (are students bored, or not?) and TIME MANAGEMENT (are students making good choices about how they spend their time, and are they spending enough time?). I’m guessing that the systems can try to do a good job of monitoring time spent, coming up with measures of student time spent on different tasks, proximity of that time to the deadline associated with a task, etc. (although D2L gives me NOTHING like that right now…) – but what about engagement? Has anybody built a system that automatically facilitates student rating of everything they do on a simple scale, with something like “Did you enjoy this assignment?” (no, sort of, yes, very much) and “Did you learn something from this assignment?” (not sure, no, maybe, yes). That’s something I would love to see, and I have not seen it anywhere.

  2. lauragibbs
    August 21, 2013

    P.S. I should add: that kind of data is not so much useful to the students, but it would be incredibly useful to me, the teacher. How can I know where to target my efforts to improve the course and course materials if I don’t know what the students think about each and every item…? And a big part of improving student success in a course is not about changing the students so much as it is about changing the course to make it better! Again, IMHO. :-)

  3. Education Cowboy
    August 21, 2013

    Michael,

    Thanks, as always, for providing such detailed and lucid information on this topic. Also, to Laura, I agree with your comment on the key to student success being engagement. Almost a decade ago, Andy Peterson and I presented a paper on environmental scanning as part of teaching online courses, and the tools we had available to measure and influence real student engagement. I find it somewhat fascinating that we have progressed so far in some ways but continue to struggle in others where this is concerned.

  4. Michael Feldstein
    August 21, 2013

    I’ll start by reposting the reply I gave to Laura on G+:

    Thanks, Laura. I almost made a comment in that post about the potential for MOOCs to be a positive influence on this dynamic, but I decided it was a bit too far off the path. I’ll save it for a different post at a different time.

    I’m not sure about the answer to your question, but frankly skeptical about self-reporting of engagement anyway. It’s generally much more effective to find behavioral evidence for that sort of thing. Discussion board participation patterns can be one indicator. D2L’s S3 product does include a social network analysis tool, but I don’t know the specifics of it. Student access of optional materials can be another indicator.

    I did write a post a while back on analytics that could measure and encourage engagement: http://mfeldstein.com/the-zone-of-proximal-curiosity/. It’s just a fantasy at this point, though.

    Rob I agree that the sort of amazing progress/utter lack of progress split is somewhat head-spinning.

  5. Chris Munzo
    August 21, 2013

    You have to have the LMS to get the embedded analytics. I would like to hear from Blackboard and D2L management about whether the analytics are key selection criteria in winning new clients and keeping existing ones. Or are they just one of many features that are equally weighted in LMS evaluations? In other words, is the inevitable increase in the base price of the LMS worth the investment?

  6. Eilif Trondsen
    August 22, 2013

    Thanks for addressing a topic that obviously gets a lot of interest these days, Michael. And perhaps in a future post, you can take a shot at analytics in a MOOC context, and perhaps review what the LyticsLab.org at Stanford–and similar “labs” that I suspect other institutions are setting up–plan to do, or hope to accomplish with all the MOOC data they hope to capture.

    I was intrigued by the contrasts you presented about the D2L and Bb analytics “world view” and approach and I am curious about how Instructure/Canvas compares to D2L and Bb in terms of what they offer and how they approach analytics.

    I assume you saw the piece by Joshua Kim the other day in IHE on Abilene Christian University and their decision to go with OpenClass. Pearson is of course very focused on analytics, and is working closely with Newton in the ASU project, but I wonder if OpenClass provides much in terms of analytics?

    I look forward to future posts and enjoy everything you and Phil write.

  7. Jon
    August 28, 2013

    Michael,

    As always, I appreciate your reporting and look forward to learning more about how all the LMS providers are approaching the Analytics topic.

    I am trying to get a grip on exactly what value D2L is bringing to the table. I have been using D2L for the past few semesters. Prior to that our institution used ANGEL for 6-7 years. To be sure, this was not a popular move at our institution due to the loss of functionality and the un-intuitive user interface many of us power-users have discovered. (I will reserve my rant about the gradebook for another day)

    From my experience, the functions provided to faculty via the D2L “Analytics Portal” do not even approach the basic reporting found ANGEL. I also use Moodle as a training platform. Again, Moodle reporting exceeds the capabilities of D2L.

    Inspired by your blog post I decided to do a test. Here is what I found:

    On the first day of class, I had 3 student emails in my D2L inbox. Yet according to the D2L Analytics reports, none of my student had logged in. I further compared the Class Progress Report with the User Course Access Report from Analytics and again found the same disconnect.

    Data integrity and validation are very important issues. If you are going to make decisions based on the data, it better be accurate. Based on these reports, I would be uncomfortable making any kind of decision that would impact my students’ grades or enrollment status. Since I am not an administrator, I cannot speak to the integrity of the data l above the “course level”, but it is safe to assume that accuracy begins at the base level.

    To be sure, Learning Analytics is becoming one of the most important issues in education. Data is a great and terrible thing depending on how it is used/interpreted.

    Here is a beautiful quote to ponder….

    “What makes a scientist great is the care that he takes in telling you what is wrong with his results, so that you will not misuse them. “ …W. Edwards Deming

    I challenge you in future reporting to take us deeper into all of the Analytics products so we can have a better understanding of their applicability.

    Than you for you continued brilliance in reporting.

    Jon

  8. Michael Feldstein
    August 28, 2013

    Thanks, Jon. We are definitely interested in reporting on the kind of deep dive product reviews you are asking for, but it is difficult from both time and access perspectives. We’ll keep thinking about how we might accomplish something like this.

  9. Phil Hill
    August 28, 2013

    Jon, I would add that what you describe fits into, and gives additional details on, Michael’s comment that “We heard some hints of data integrity problems as well, but nothing too specific or definitive” (from D2L Analytics Update post).

    You also make great points that “Data integrity and validation are very important issues. If you are going to make decisions based on the data, it better be accurate.” followed by “Data is a great and terrible thing depending on how it is used/interpreted.”

Back to top
mobile desktop