Measure What You Value: Mark Glynn and Juha Martikainen on Trends in Learning Analytics
IntelliCon 2021 gave us the exciting opportunity to host a roundtable discussion with world-renowned Learning Analytics experts Mark Glynn, Head of Teaching Enhancement at Dublin City University (DCU), Ireland, and Juha Martikainen, Solution Owner at a top-rated Finnish University.
Mark Glynn has a Ph.D. in Chemistry, an MSc in e-learning, and certificates in learning and teaching in higher education and online assessment. He has deep insight into flexible learning, learning virtual learning environments, and many other innovative approaches to practices in higher education. He currently works with staff at DCU on both technological and pedagogical-based projects to enable the implementation of flexible learning.
Juha Martikainen is a learning environment specialist with experience in e-learning projects, collaboration tools, web design, and teaching. He is currently developing the digital learning environment and associated services ecosystem in Finland, fostering close collaboration between the scientific, business, and arts communities to facilitate multidisciplinary education and research.
We talked about improving course design through analytics and teacher buy-in, retaining pandemic era data-gathering practices, measuring what matters most, and making data actionable and easy to access. Here are some highlights.
Student, Teacher, and Institution-wide Analytical Lenses
What is the reality of Learning Analytics now at your institutions?
Mark Glynn (MG): “At DCU, we have been looking at analytics for a long time. We have been analyzing the data from the students: where they’re from, how long they stay in the institution, what resources they use, and it helps us in making informed decisions on where we’re going. Specifically, with Learning Analytics, we have been experimenting for several years. The earliest project I can think of is 2013. But the interest has varied from an institution level or right down to an individual classroom level.”
Juha Martikainen (JM): “Well, it’s kind of similar. We are actually three institutions in one, and all of these schools have done some kind of Learning Analytics before in the courses with teachers doing it or students asking for it. But only in the last few years [have] we actually started to do or piloted a coherent solution and not a school-based solution. Some of our schools are very forward-thinking. They did student-level and course-level analytics for years now. The latest development has been to look at the analytics in two different ways, one of them being the university-level, like a governor’s level (such as How many students are succeeding in their studies and so forth.); the other one is that we’re trying to help the students and teachers. The latest trend is to support the students in their studies.”
Analytics as a Tool to Inform Course Design
Do you think that there is a tendency that teachers will be looking at Learning Analytics after the pandemic? How engaged are the teachers with analytics?
MG: “More teachers are interested in data. They would like to have evidence that they’re doing well, evidence of areas where they need to improve. So I’m delighted to say that the appetite for analytics got better and better. The challenge is to make sure they’re using the right data, that they construct their course in a certain way to get the best data that they can. Because what you do not want to do is to give poor data to students or the staff because if they are making bad decisions on the basis of this bad data, it can actually have a really negative effect. So for me, it is a careful balance that we must take when we’re promoting Learning Analytics, that we’re also promoting proper course design to ensure that the best experience is obtained from this data.”
JM: “Mark hit the nail on the head because the biggest problem is the course design and the information we get from the courses, and that’s the first thing that should be done before building into more technical things. There is more and more need for Learning Analytics from teachers. (How could I do my course better?) One of our main goals is to get the teachers interested about what their courses are like, how the courses are doing, how the students are doing in there. They would actually create better courses, and I think analytics could help them in that.”
Analytics in the New Normal: Measure What You Value, NOT Value What You Measure
As an analytics provider, we noticed a huge increase in the search of analytics during the pandemic. Can you share more about your pre and post scenario of the pandemic?
JM: “I don’t think it was easy for anyone, but we were in a good place because we already had a running learning management system [LMS], we had running integrations, and we had integrated Zoom and [other] services as well and we had started to delve into analytics before the pandemic, so we wanted to use these services as efficiently as possible, so it wasn’t that big of a leap to change into.”
MG: “Good points is that more people engaged with the VLE [LMS]. From an analytics point of view, that would mean there is more data. What I would be interested in is in 12-months time how many people that have moved to the VLE [LMS] are still engaging with the VLE [LMS]. And also, when they move, what activities they are choosing and how were their courses set up and how would they look for better flexibility to students, to me this would be the really interesting data. Now, as we are returning to normal, what good practice are we going to continue, and Learning Analytics helps us identify an awful lot of good practice and bad practice as well.”
JM: “That’s actually a really good point because that’s very interesting to see what kind of practices are taking forward even in the new normal time. And that’s actually a difficult way or difficult thing to measure because measuring good teaching is difficult when you are looking into technical stuff.”
MG: “I use a phrase where you should ‘measure what you value’ NOT ‘value what you measure’ it’s important even though the words are the same but in a different order. And if we value good teaching, we should come up with a way to measure good teaching to see when it happens. To me, student engagement is a good indicator. Every click they take is not maybe the best indicator, but it is one piece of the jigsaw, and it’s important to recognize that it is only one piece and that you can pull data from all different sources.”
Actionable Analytics with IntelliBoard
Can you expand on how useful data could be for you?
MG: “For me, one example is: we moved our orientation online (over 4000 students joined us), we were able to measure their clicks in the VLE [LMS]. We were able to prove the benefits of going online through our data. We were able to prove that before our students took a step into the campus, they were engaged with the university 24 hours a day, seven days a week, whereas the traditional face-to-face orientation typically took place Monday to Friday, sometime between 9 o’clock and 5 o’clock. And this is where IntelliBoard helped us because it made the data easy-to-see. The data is always there and connected to different services, in our case, to Moodle. [This sort of data allows me to show university management the impact of what we are doing and the benefits of the approaches we take.]*”
JM: “This is the same for me. I’m showing the number of hits to get more server power. Or let’s say this is the number of Zoom meetings we had. Can we get an upgrade?”
MG: “And then on the local (or classroom) level, we’re able to use the data to show the staff how many times the resources were viewed. It allows them to reflect on how to make a better video to be more engaging for students. With IntelliBoard, we can show who is completing the course and who is not completing the activities because it is very valuable for us, and [it could be very important] for various organizations to improve student retention and analytics gives us the ability to do that.”
Trends in Learning Analytics
Both institutions had already set up the technology necessary to use Learning Analytics before the pandemic. Still, the pandemic’s onset accelerated the adoption of these practices at the student, teacher, and institution-wide levels. This increase highlighted the power of Learning Analytics as well as its potential pitfalls. Thinking through course design in tandem with Learning Analytics is critical, so you don’t wind up making decisions off bad or haphazard data; you need to focus on what really matters to you and intentionally set up analytics to measure it. Being able to access and visualize that data easily is also vital because that’s what makes the data actionable. Tools like IntelliBoard are crucial to closing the loop and connecting Learning Analytics back to supporting learners themselves in a meaningful way.
Thank you for the great conversation, Mark and Juha! We are so grateful to have had you at IntelliCon and to be able to help you put your analytics into action.
*We spoke further with Mark Glynn after his IntelliCon RoundTable and learned more about how he puts Learning Analytics into action. Thank you so much, Dr Glynn!