Measuring Social Impact of Education Technology
In the last decade, educational technology a.k.a. edtech has become a very popular buzzword with a lot of high hopes. But still, most of the analysts describe the education sector as one of the last major industries that have not been disrupted by the digital world. In this blog post, I’ll tell you about my MSc thesis on how we can understand the social impact of technological solutions in education. This short summary is based on a literature review and some field data coming from investors and other practitioners.
Here is the main picture first: We have an edtech solution targeted to a specific group of people which aims to create some kind of positive change. Using this simple process as a base, I found 5 core components to measure the social value of this particular edtech solution.
The first component is scale which might be one of the easiest to measure. This component is about understanding how many people this solution is reaching. In other words, it is the total number of end beneficiaries of a particular solution.
The second component is also somehow related to the scale component which is about the ratio of the underserved population. In most of the cases, this component is a measure of socio-economic equality. So, we try to understand how much of these people are underserved. Here, it is important to clarify the definition of underserved. For example, in some cases, we can measure the ratio of women regardless of their socioeconomic status -let say if it’s an app that teaches coding. In other cases, it might be about people with learning disabilities and so on.
The third component that we can look at is product quality. A poorly designed educational solution can create even negative impact on the learner. Let’s say we have a language learning solution which doesn’t work properly, doesn’t give good quality feedback, too difficult or too easy, or the user experience is just very poor. This can easily influence the learning motivation in a negative way. That’s why it’s very important to understand if the product/service we are measuring is based on sound pedagogical principals and it is technologically feasible with reasonable user experience.
Let’s jump to the fourth component: implementation quality. According to US-based research, 95% of the purchased products in schools has not been utilized fully. More surprisingly, 37% of the licenses have not been even activated. If the solution is not used, it doesn’t create an impact. So, measuring user engagement is very important. In addition to that, we need to understand the contextual factors that affect the implementation quality. Do we need high-speed internet? Can we use iPads only, or does it work with any devices? What level of technological expertize is it required from the teacher for this solution? And the list goes on.
The last component I will mention might be one of the most difficult ones to measure. It is about the contribution of a particular solution to our intended impact goal. In other words, it is the depth of impact. Let’s say we have a learning app focusing on math. But in what level does this app helps the students? Does it improve the learning outcomes from 50 to 52 or from 50 to 100? So, how deep the impact of this particular solution? In this component, one of the important limitations is the quality of learning assessment. If we are measuring math scores, we can claim a certain level of reliability based on some standardized test. But when it comes to improving student well-being or learning motivation it becomes much more difficult to measure the depth in a reliable way.
Overall, those are the 5 components I found out during my thesis. As almost all of the practitioners and academics, I also think that this is not easy to measure. In some cases, it might be even close to impossible to come up with a standardized and very reliable measurement model. With that said, this is an ongoing effort to improve what we have to create a better future. As the famous saying goes, you can’t manage what you can’t measure.
If you want to read the full text of my thesis or just have a chat about this topic, you can send me an email to semih.ersoz@aalto.fi!
Aalto Entrepreneurship Platform
