Achievements and challenges in learning analytics in Spain: The view of SNOLA
Logros y retos en analítica del aprendizaje en España: La perspectiva de SNOLA
Achievements and challenges in learning analytics in Spain: The view of SNOLA
RIED. Revista Iberoamericana de Educación a Distancia, vol. 23, núm. 2, 2020
Asociación Iberoamericana de Educación Superior a Distancia
Recepción: 24 Enero 2020
Aprobación: 27 Febrero 2020
How to reference this article: Martínez-Monés, A., Dimitriadis, Y., Acquila-Natale, E., Álvarez, A., Caeiro-Rodríguez, M., Cobos, R., Conde-González, M. A., García-Peñalvo, F. J., Hernández-Leo, D.,
Menchaca, I., Muñoz-Merino, P. J., Ros, S., y Sancho-Vinuesa, T. (2020). Achievements and challenges in learning analytics in Spain: The view of SNOLA. RIED. Revista Iberoamericana de Educación a Distancia, 23(2),
pp. 187.212. doi: http://dx.doi.org/10.5944/ried.23.2.26541
Abstract: As in other research fields, the development of learning analytics is influenced by the networks of researchers that contribute to it. This paper describes one of such networks: the Spanish Network of Learning Analytics (SNOLA). The paper presents the research lines of the members of SNOLA, as well as the main challenges that learning analytics has to address in the next few years as perceived by these researchers. This analysis is based on SNOLA’s archival data and on a survey carried out to the current members of the network. Although this approach does not cover all the activity related to learning analytics in Spain, the results provide a representative overview of the current state of research related to learning analytics in this context. The paper describes these trends and the main challenges, among which we can point out the need to adopt an ethical commitment with data, to develop systems that respond to the requirements of the end users, and to reach a wider institutional impact.
Keywords: groups and organizations, data processing, trend.
Resumen: Tal y como ocurre en otros campos de investigación, el desarrollo de la analítica del aprendizaje está influido por las redes de investigadores que contribuyen al mismo. Este artículo describe una de estas redes: la Red Española de Analítica de Aprendizaje (SNOLA). El artículo presenta las líneas de investigación de los miembros de SNOLA, así como los principales retos que la analítica del aprendizaje tiene que afrontar en los próximos años desde la visión de estos investigadores. Este análisis está basado en datos de archivo de SNOLA y en una encuesta realizada a los actuales miembros de la red. Aunque esta aproximación no cubre toda la actividad relacionada con analítica del aprendizaje en España, los resultados proporcionan una visión general representativa del estado de la investigación relacionada con analítica del aprendizaje en dicho contexto. El artículo muestra cuáles son estas tendencias y los principales retos, entre los que se encuentran la necesidad de adoptar un compromiso ético con los datos, desarrollar sistemas que respondan a las necesidades de los usuarios y alcanzar mayor impacto institucional.
Palabras clave: grupos y organizaciones, procesamiento de datos, tendencia.
Research in Information Science, as well as in many other disciplines, is enacted by intertwined networks of people and technology (Latour, 2005). Learning analytics (LA) as a research field illustrates this idea well, as it has witnessed a fast development, partially due to the intense work of different networks of researchers that have sustained it. The society for Learning Analytics Research (SOLAR) (Solar, 2020) is the most influential, bringing together researchers and actors related to LA across the globe. While creating SOLAR and with the aim of fostering research on LA at more local levels, several networks were founded at the beginning of the 2010’s in different regions. Some of them started out as informal organizations of people, whereas in some cases they were created on the basis of a founded project. One example of these officially recognized networks at the European level is the Learning Analytics Community Exchange (LACE) (LACE, 2020), which contributed with some outstanding work between the years 2015 and 2017 and is still active in connection with SOLAR.
In Spain a group of researchers working in the LA field formed the Spanish Network of Learning Analytics (SNOLA) (SNOLA, 2020) in 2013. This network was officially recognized in 2015 by the Spanish Government. In 2020 SNOLA has started a new period of activity as an officially recognized thematic network, with new groups that add more variety and quality to the network. Although SNOLA does not represent all the work being done in LA in Spain, it comprises a wide representation of groups. The description of the work done by its members provides a good picture of what is being done in LA research in Spain, in line with recent works providing a more general overview of the field (Papamitsiou, Giannakos, & Ochoa, 2020).
Scientific networks on specific areas can be developed at different levels, e.g. global/regional/national, or coming from other existing networks and societies. The analysis and study of the networks at different levels, origins or focuses may be relevant in order to understand the trends and challenges that are in the center of attention of these networks. This is especially relevant for the field of LA that emerged in the last ten years as a consequence of the interest of a number of researchers that were working in related research areas (i.e., Artificial Intelligence in Education, Computer Supported Collaborative Learning, Educational Data Mining, or Technology Enhanced Learning), each one of them with its own community behind. Although some studies have been undertaken in order to provide an overview of the global landscape in LA (Ferguson et al., 2016), it is still important to further analyze and understand the evolution, trends and challenges that are detected in an active network at a finer grain level (national).
This paper presents SNOLA: its motivation, goals, activities, and expected outcomes in the following years. It provides an overview of the work carried out by its members, giving a partial but meaningful panorama of current research and challenges of LA in Spain. This overview is based on archival information about the activity of the network in the past years and its working plan for the future. As a second contribution, the paper presents an outline of the challenges that should be faced by the LA research field in the next few years. The description of the activity of the groups and their vision of the challenges was collected through an open-ended survey that was filled out by the representative of each group in the network. Using this survey, we have collected the main research lines of each group related to LA, as well as the participants’ view of the main challenges of the LA research field in the near future.
The structure of the paper is as follows. First, we describe SNOLA, its motivation and main goals, the activities it has carried out in the past and what impact is expected from the network in the future. Then, the research groups belonging to SNOLA are introduced. Third, we describe the work done by its members in the last 5 years, followed by a summary and discussion of the main challenges for LA described by its members.
SNOLA – MOTIVATION AND ACTIVITIES
Education is a key element in our society. Institutional reports by the European Commission (EC, 2016) have pointed out the potential of LA for the improvement of the quality of the European education system, but also the fact that many research questions and technical issues need to be solved to reach this goal. The need to face these issues has led the European Union to make a series of recommendations for the field of LA (Ferguson et al., 2016). One of these recommendations is the need for structured organizations to support the use of LA; the urgency of bringing together all relevant stakeholders to collect and analyze the evidence about the present and expected benefits of LA; and to identify the needs of the implied actors, especially students and teachers.
SNOLA appeared with the main aim to face these demands. It was established informally in 2013 by Spanish researchers interested in the interchange of knowledge and expertise in the area of LA. Its mission was to create synergies and explore possibilities of collaboration among them, as well as becoming a reference in the field of LA in Spain. SNOLA also aimed to promote the connection between Spanish researchers with other international networks in different regions of the world, including Latin America, Australia, US and Europe. The work of the network was reinforced by its official recognition as a Thematic Network of Excellence by the Spanish Government in 2015 and in 2020.
One of the main activities of the network since its creation has been the organization of scientific events that have served to disseminate the work of the network and to extend the connections to other researchers, both in Spain and at an international level. Since 2013, SNOLA has organized numerous events: the series of Learning Analytics Summer Schools, (LASI Spain) which have taken place annually since its first edition in 2013, in Madrid, up to its 2019 edition in Vigo ( Caeiro-Rodríguez, Hernández-García, & Muñoz-Merino, 2019).
These networking activities have helped to extend the network to other forums, such as the events organized by the e-Madrid network, the organization of the special track on LA in the TEEM conference from 2013 (Conde-González & Hernández-García, 2013) to 2019 (Conde-González & Hernández-García, 2019), and the organization of the workshops WLA-CISTI and WAPLA. This intense activity has been possible due to the existence of the network itself. The communication and coordination channels set up by the SNOLA have facilitated to build up working teams to organize these events in a smooth and efficient fashion, which would have been not possible without the existence of the network.
The network has maintained contact with relevant researchers in the field of LA at the international level, who have offered webinars organized by the network, or have participated as keynote speakers at different editions of LASI and the special tracks on LA organized by SNOLA. These connections have materialized as well in the participation of some of the members of the network in international projects. For example, in Europe, the project SHEILA (Tsai et al., 2018) has worked in the deployment of LA policies at an institutional level. This project has had a continuation in Latin America with the project of Learning Analytics in Latin America (LALA, 2020)
Finally, it is interesting to note that the number of organizations (companies, public administrations, universities, educational institutions in general) and persons interested in the network goes beyond the list of the official members of SNOLA. The SNOLA webpage (SNOLA, 2020) has become a hub for information about LA in Spain, with news, resources, software, and publications that provide a good overview of the current state of research in the area in our context.
Besides the networking and dissemination activities, the value and richness of the network is derived from the activity of its members. This section describes the groups that belong to SNOLA and the main research lines in which they are involved.
SNOLA, as an officially recognized thematic network, comprises 12 research groups, all of them belonging to different universities across Spain. These groups are summarized in Table 1, which includes a link to their URL. As it can be observed, most of its members come from a technological background, from pure Information Technology (IT)-related areas like Telematics Engineering, Computer Languages and Systems, Artificial Intelligence, and Computer Architecture; or applied areas, such as Didactics of Mathematics or Business Organization. Therefore, SNOLA as a whole has been traditionally more focused on technology-oriented contributions rather than other aspects which are also important (e.g. pedagogical, philosophical, etc.). This has been also the case with the wider LA field at an international level. However, the community is fully aware of the need to consider the ethical, social, and pedagogical issues raised by the domain in which these technological contributions are presented. The increasing weight of these issues in current discussions in the field points out to the need of a more interdisciplinary approach in the composition of the groups and of the network. A challenge of SNOLA is to attract researchers that come from social and pedagogical fields to collaborate closely with its members, in order to face the more global challenges posed nowadays in LA. These challenges are discussed more deeply in the following section.
Table 1 evidences a considerable experience in LA of the members of SNOLA, given the range of years in the field (from 5 to 20 years). Some of these groups were already working in research problems that are now considered as part of LA years before the field was recognized as a separate research area with this name. This is explained by the fact that most of the groups that belong to SNOLA joined the specific LA field from a broader research field related to the use of computers in education, like e-learning, Technology Enhanced Learning (TEL), Computer Supported Collaborative Learning (CSCL), Educational Data Mining (EDM) or Artificial Intelligence in Education (AIED).
|GROUP||URL||Knowledge Area||University||Years in the area|
|BigDataLab||http://bigdatalab.scc.uned.es||Architecture and Computer Technology||UNED||5|
|Deusto LearningLab||http://dtlearning.deusto.es||New technologies applied to Education||UDeusto||7|
|Group for Adaptive Teaching-Learning Environments (GaLan)||https://galan.ehu.eus/Galan||Computer Languages and Systems||UPV/EHU||8|
|Grupo de Herramientas Interactivas Avanzadas, Advanced and Interactive Tools Group (GHIA)||http://vghia.ii.uam.es/ghia||Computer Science and Artificial Intelligence / Computer Languages and Systems||UAM||7|
|Telematics Systems and Engineering (GIST)||https://bidi.uvigo.es/es/grupo/gist-grupo-de-ingenieria-de-sistemas-telematicos||Telematics Engineering||UVigo||7|
|Gradient Lab – Group of Telematic Applications and Services (GAST)||http://gradient.gast.it.uc3m.es||Telematics Engineering||UC3M||17|
|GRoup of Interaction And e-Learning (GRIAL)||http://grial.usal.es||Computer Science and Artificial Intelligence, Computer Languages and Systems, Research Methods and Assessment in Education||USal||11|
|Grupo de Robótica - Robotics Group||http://robotica.unileon.es||Architecture and Computer Technology, Computer Languages and Systems, Computer Science and Artificial Intelligence,||ULeón||6|
|Intelligent and Collaborative Systems Group (GSIC)||http://www.gsic.uva.es||Computer Languages and Systems, Telematics Engineering, Didactics and School Organization||UVa||20|
|Learning Analytics for Innovation and Knowledge Application in Higher Education (LAIKA)||http://laika.blogs.uoc.edu||Online mathematics education, Computer Science||UOC||6|
|Research Group on Interactive and Distributed Technologies for Education (TIDE)||https://www.upf.edu/web/tide||Telematics Engineering, Computer Science, Cognitive Sciences||UPF||6|
|Tecnologías de la Información para la Gestión Empresarial (TIGE) – IT for Business Management||http://tige.ior.etsit.upm.es||Business Organization, Computer Science||UPM||11|
SNOLA LINES OF RESEARCH
In order to provide an overview of the work done by the network, the groups reported their research lines in an internal survey, which was inspired by the classification of LA systems presented in Omedes (2018). A total of 34 distinct research lines were identified. For each research line, the following data has been collected: brief description; who is the main user of the system/functionality; educational level to which it has been applied/thought of; data sources and analysis techniques; the type of system (whether it is descriptive or prescriptive); and the main expected benefits of the system. Regarding this feature, we started with the four possible values suggested by Omedes (2018), and added three additional ones that emerged from the data provided by the partners. This set of categories are shown in Table 2, with a description of their meaning, the code we will use to refer to them in this section, and the number of research lines that were reported to fall under each research goal.
|Code||Expected benefit||Description||Num of lines|
|ILRP||Increase learner retention and performance||Reduce dropout rates and increase students’ performance. Having the right insights allow for performing proactive tutoring and intervention.||26|
|IQCCLE||Improve the quality of the content, the course or the learning environment||Discover content consumption patterns, understand content quality issues, and provide personalized learning experiences (adaptive learning).||17|
|PDS||Proactively drive success||Identify and promote success factors as well as understand students’ pathways leading to graduation (curriculum design).||11|
|ACE||Allocate costs effectively||Help in discovering which resources work and which don’t. Selective investment strategies may be better designed based on our analytics.||3|
|IIL||Identify indicators for learning||Define proxies for learning based on data.||7|
|CBLM||Create better learner models||Identify elements of the learner model.||4|
|PRFA||Promote regulation, formative assessment of self-reflection about the learning process||Show elements of learning to the learner (or teacher) and enable their reflection about learning.||3|
The data reported by SNOLA members has been classified according to their similarities in a number of categories. This classification is not orthogonal: some of the research lines might belong to different categories, but we present them in only one of them, to make the description more clear and simpler to follow.
Predictive LA systems
The first category is composed of predictive LA systems. Prediction of learning variables, such as students’ performance or students at risk is one of the most well-known functionalities of LA applications (Peña-Ayala, 2018). It is therefore not surprising that it is the most frequently reported research line, with six lines (see Table 3). Some of these lines focus on the prediction of learning results and dropout in MOOCs ( Moreno-Marcos et al., 2020; Cobos & Olmos, 2018), or on-line and blended courses (Martínez, Campuzano, Sancho-Vinuesa, & Valderrama, 2019; Agudo-Peregrina, Iglesias-Pradas, Conde-González, & Hernández-García, 2014). In general, these systems aim at deriving high-level indicators from the low-level data provided by the system logs (Alexandron, Ruipérez-Valiente, Chen, Muñoz-Merino, & Pritchard, 2017). One of such indicators is academic engagement (Bote-Lorenzo & Gómez-Sánchez, 2018). The users of these systems are students and teachers, who are expected to benefit from actionable information provided by the predictive systems. In some of the cases they also address managers in charge of the institutional learning environment, or researchers that aim at identifying indicators for learning (IIL) based on the low-level data provided by the system. In general, works related to prediction need a certain amount of data. For this reason, they are frequent in virtual learning contexts at scale, enacted on LMSs (like Moodle) or MOOCs, although other environments like version control systems are considered as well (Guerrero-Higueras, DeCastro-García, Rodríguez-Lera, Matellán, & Conde, 2019). As it can be expected, these research lines use different versions of predictive analysis techniques from artificial intelligence and machine learning. The main expected benefits from these research lines are to improve learner retention and performance (ILRP) as well as to improve the quality of the course content and the learning experience (IQCCLE).
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Expected Benefit|
|Prediction of learning results and dropout||(Moreno-Marcos et al., 2020)||S / T / M||Students’ actions on the system (MOOC)||Random Forest, Regression, Neural Networks, Decision Trees||P||ILRP, IQCCLE|
|Prediction of learning results and dropout||(Cobos & Olmos, 2018)||T / M||Students actions on the system (MOOC)||Predictive analytics, Machine Learning, Statistical analysis||P||ILRP, IIL, CBLM, PRFA|
|Identification of engineering students at risk||(Martínez et al. 2019)||S / T||Students actions on the system (Moodle and Virtual Campus)||Predictive analysis||P||ILRP, PDS|
|Actionable information based on prediction of academic engagement in MOOCs||(Bote-Lorenzo & Gómez-Sánchez, 2018)||S / T||Students’ actions on the system (MOOC)||Feature selection, Machine Learning||P||ILRP, IQCCLE|
|Analysis and classification of student interaction data with prediction purposes (Interactions)||(Agudo-Peregrina et al., 2014)||T / M / R||Student Activity (Moodle log data)||Log data classification, Regression||P||ILRP, IQCCLE, IIL|
|Educational data mining||(Guerrero-Higueras et al., 2019)||S / T||Students actions on the system (version management system)||ML||P||ILRP, IQCCLE|
|Definition of high-level actionable indicators based on low level data.||(Alexandron et al., 2017)||S / T||Students’ actions on the system (MOOC)||ML, Artificial Intelligence Techniques, Semantic modelling, Heuristics||P||ILRP, IQCCLE|
A second category of research lines frequent in SNOLA is visual analytics (see Table 4). The focus on data visualization is also reported by Peña-Ayala (2018) as a main functionality of LA systems. In SNOLA, the groups that work in this research line aim at providing dashboards oriented to teachers (Gómez-Aguilar, García-Peñalvo, & Therón, 2014), students (Tobarra et al., 2014) or both (Ruipérez-Valiente, Muñoz-Merino, Leony & Delgado Kloos 2015). These systems are generally aimed at providing dashboards with indicators derived from the log data, but some of them are oriented to more specific purposes, like the one by Chaparro-Peláez, Iglesias-Pradas, Rodríguez-Sedano and Acquila-Natale (2019), which aims at supporting peer- and self-assessment processes. Most of these systems have a descriptive nature, although some others combine descriptive and prescriptive objectives (Tobarra et al., 2014). Sometimes these systems are also designed with the aim of helping managers allocate costs effectively (Cobos, Gil, Lareo, & Vargas, 2016), besides other outcomes, such as helping teachers identify indicators for learning. These systems use techniques specific to visual analytics, sometimes supported by heuristics or descriptive statistics. Some of the reported research lines are currently focused on frameworks to facilitate the automatic generation of visualizations, like the system by Vázquez-Ingelmo, García-Peñalvo, and Therón (2019), to generate dashboards adapted to the needs of different types of users, or the one by Hernández-García and Suárez-Navas, (2017) oriented to the creation of graph data based on social network analysis.
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Expected benefit|
|Visual analytics of eLearning systems (VeLA)||(Gómez-Aguilar et al., 2014)||T||Students’ actions on the VLE, Grades||Visual analytics||D||ILRP|
|LA Dashboards for virtual labs||(Tobarra et al., 2014)||S||Platforms logs||Heuristics||D/P||ILRP, PDS|
|Visual Analytics of students’ actions||(Ruipérez-Valiente, et al., 2015)||S / T||Students’ actions on the system (MOOC)||Visual analytics||D||ILRP, IQCCLE|
|LA Dashboard for MOOCs||(Cobos et al., 2016)||T / M||Students’ actions on the system (MOOC), grades, demographics, self-reported data||Descriptive Statistics||D||PDS, ACE, IIL|
|Visualization of peer and self-assessment data in Moodle (MWDEX)||(Chaparro-Peláez, et al., 2019)||T||Peer-assessment grades (Moodle Workshops)||Visual Analytics||D||ILRP, IQCCLE, IIL|
|Automatic generation of adapted dashboards||(Vázquez-Ingelmo et al., 2019)||S / T / M / R||Importar lista||Multi-Dimensional Analysis (MDA), Machine Learning (ML)||D||ILRP, IQCCLE, PDS, ACE|
|Graph generation of educational data in online learning forsocial network analytics (GraphFES)||(Hernández-García & Suárez-Navas, 2017)||T / M||Student activity (Moodle log data-Forums)||Social Network Analysis, Data visualization||D||ILRP, IQCCLE, IIL|
Support to active learning strategies
The systems in the previous categories are based on distance learning and/or MOOCs, but LA is also used to promote other types of learning. Table 5 describes the lines of research related to this goal. These types of learning include collaborative learning (Amarasinghe, Hernández-Leo, & Jonsson, 2019); adaptive learning (Muñoz-Merino, Novillo & Delgado Kloos, 2018); peer feedback (Er, Dimitriadis, & Gaseviç, 2019); social learning (Claros, Cobos, & Collazos, 2015), and flipped classrooms (Rubio-Fernández, Muñoz-Merino, & Delgado Kloos, 2019). All these proposals share their orientation for both teachers and students. In general, they are expected to improve learner retention and performance (ILRP); to proactively drive success (PDS), by analyzing the paths that make the students perform better and using this knowledge to make recommendations to the users; and at improving the quality of the course content and the learning experience (IQCCLE). In many cases, the results of the LA system are used to improve the learning design (LD) of the courses. This link between LA and LD is gaining momentum in the field of LA and may have higher impact in the upcoming years.
A slightly different approach related to this category is the one taken by Manso-Vázquez, Caeiro-Rodríguez and Llamas-Nistal, (2018) that aims at defining design criteria for self-regulated learning, including the definition of the traces that should be generated by these systems according to the xAPI specification.
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Goal|
|Orchestration of collaborative learning activities (PyramidApp)||(Amarasinghe, et al., 2019)||S / T||Actions on PyramidApp: progress in the activity, answers to the tasks, students’ discussions||ML, descriptive statistics, data visualization||D P||ILRP PDS|
|Adaptive learning based on user models||(Muñoz-Merino et al., 2018)||S / T||Students’ actions on the system (Intelligent Tutoring Systems)||Bayesian networks, rules, Item Response Theory.||P||ILRP, IQCCLE|
|Support to dialogic peer feedback (Synergy)||(Er et al., 2019)||S / T||Students actions on the system, content of the feedback,||Descriptive statistics||D||ILRP|
|Social learning supported by learning analytics||(Claros et al., 2015)||S / T||Students actions on the system (content and social)||SNA, CSCL||D||IQCCLE, PRFA|
|Learning analytics to improve Flipped Classrooms||(Rubio-Fernández et al., 2019)||S / T||Students’ actions on the system (SPOC)||Visual analytics, clustering, adaptation for improving the flipped classroom||D||ILRP IQCCLE|
|Definition of design criteria for self-regulated learning support tools||(Manso-Vázquez, et al., 2018)||M||xAPI profile||-||D||CBLM|
Learning analytics for Learning Design
The fifth category recognizes the importance of the relation between learning design and LA in a different direction: using LA to support LD processes. This is the common goal of the two research lines reported in Table 6. The system by Michos, Hernández-Leo and Albó (2018) uses the actions on a social network for teachers to support learning design processes, while the line reported by Wiley, Dimitriadis, Bradford and Linn (2020) aims at developing a framework for developing and evaluating learning analytics for learning design.
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Goal|
|Support to learning design processes (ILDE2)||(Michos, Hernández-Leo, & Albó, 2018)||T||Actions on ILDE2, (a kind of social network for teachers), feedback on teachers’ and students||Social Network Analysis (SNA), data visualization, descriptive statistics||D||IQCCLE|
|Learning analytics for learning design (OrLA, T-Glade, TAP)||(Wiley, Dimitriadis, Bradford, & Linn, 2020)||T / R||Students actions on the system (WISE science inquiry system); submission of results; grades||TAP (an NLP method)||D||ILRP IQCCLE|
Assessment is a central aspect of learning. In response to this fact, some of the groups in SNOLA have devoted their work to this topic. The work by Villamañe, Larrañaga and Álvarez (2017) aims at supporting these processes by providing indicators to assist evaluators in adjusting their grades. Others have focused on supporting the assessment of 21st century skills (Menchaca Sierra, Guenaga, & Solabarrieta, 2018), some of them with a focus on workgroup assessment ( Tobarra et al., 2017; Conde, Colomo-Palacios, García-Peñalvo, & Larrucea, 2018; Hernández-García, Acquila-Natale, Chaparro-Peláez, & Conde, 2018).
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Goal|
|Definition and adjustment of assessment processes (Ramon / TEA)||(Villamañe et al., 2017)||S / T / ID||Students’ answers, grades||Statistics, Regression, NNLS, Data visualization||P||IQCCLE, PDS, ACE|
|Learning analytics for the assessment of 21st-century skills||(Menchaca et al., 2018)||S / T||Grades||Heuristics||P||PDS|
|Analysis of Moodle logs for decision making and workgroup assessment||(Tobarra et al., 2017)||S / T||MOOC platform logs||Heuristic||D||ILRP, IIL, CBLM|
|Workgroup assessment||(Conde et al., 2018)||S / T||Students’ actions on the system (VLE)||Quantitative analysis and heuristics||D||ILRP, PDS|
|Measurement and analysis of teamwork indicators in online education (TeamworkRM)||(Hernández-García et al., 2018)||T||Students’ actions (Moodle log data-Forums & wikis)||Data classification (ETL), Regression||DP||ILRP, IQCCLE, IIL|
Multimodal and contextual data
One of the challenges in LA is to improve the quality of the data used to derive indicators. In this order, one category of the research works reported in this study is devoted to the definition of new data sources for LA (see Table 8). These works explore the use of human-generated data like grades (Villamañe, Alvarez, & Larrañaga, 2020) or information provided by teachers in the learning design (Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, & Dimitriadis, 2015); biometric signals captured with sensors ( de Arriba-Pérez, Caeiro-Rodríguez, & Santos-Gago, 2018); or multimodal data including both self-reported and sensor-based data (Vujovic & Hernández-Leo, 2019). Some of these works are exploratory, and are aimed at identifying indicators for learning based on new data sources or at defining better learner models.
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Goal|
|Students monitoring in blended learning environments (CASA, AdESMuS)||(Villamañe et al., 2020)||S / T||Grades||Statistics, Linear Regression, Data visualization||D||ILRP, PDS|
|Multimodal learning analytics of f2f collaborative learning||(Vujovic & Hernández-Leo, 2019)||T / R||Multimodal data, motion capture, EDA, sound, students’ self-reported data||ML, statistic analysis||D||IQCCLEIIL|
|Use of wearables to estimate levels of stress and sleep quality.||(de Arriba-Pérez et al. 2018)||S||Biometric signals||ML||D||CBLM|
|Design-aware learning analytics (GLUE!-CASS, Glimpse)||(Rodríguez-Triana et al. 2015)||T||Students actions on the system (DLE), data from the learning design, self-reported data||Heuristics||D||ILRP, PRFA|
Emotion and sentiment analysis
One of the ways to design improved learner models is the capability to model emotions. A group of research lines is devoted to this goal (see Table 9). For example, some of the works reported perform sentiment analysis on data taken from the posts of students in forums (Ros et al., 2017) or the actions of the students in MOOCs and MOOC contents (Cobos, Jurado, & Blázquez-Herranz, 2019). Another approach is the one taken by Ruiz, Urretavizcaya, Rodríguez and Fernández-Castro (2018) who modeled emotions based on self-reported data.
|Research line||Publication||User(s)||Data sources||Analysis techniques||Type||Goal|
|Social and sentiment analysis||(Ros et al., 2017)||S / T||Forum messages||Heuristics||D||ILRP|
|Academic success prediction based on emotion modelling (PresenceClick)||(Ruiz et al., 2018)||S / T||Sensors, self-reported emotions||Transition matrix, Decision trees, Data visualization||P||ILRP, PDS|
|Sentiment Analysis||(Cobos et al., 2019)||T / M||Student. actions on the system (MOOC), MOOC contents||Descriptive analytics, Natural Language Processing (NLP), Sentiment Analysis||D||ILRP, IQCCLPDS|
A final research line is related to the definition of frameworks for the adoption of LA, enacted by the participation of one of the groups in the projects SHEILA (Tsai et al., 2018) and LALA. Both projects deal with one of the challenges posed to LA at a national and international level: institutional adoption of LA. The following section deals with this and other challenges for LA in the context of SNOLA.
MAIN CHALLENGES TO LA IN SPAIN FROM THE PERSPECTIVE OF SNOLA
One of the missions of SNOLA is to identify the research challenges that should drive the attention of the research and development in the field for the next few years. In the survey used for this study, the members of the network were asked to identify the main challenges faced by the LA field. A total of 36 challenges were identified by the respondents. The answers were analyzed and clustered around main topics. This section provides a brief summary of their responses.
|Type of challenge||Frequency|
|Ethical, privacy and security issues||7|
|Increase adoption by end users||8|
|Quality of the process and the results||6|
|Personalization / Adaptation / Interoperability||5|
|Apply LA at an institutional level||5|
SNOLA has played a main role in the promotion of LA in Spain since its foundation in 2013, through the activity carried out by the network, including the organization of conferences and LA-related events, as well as the contributions of each member group to the field. The review to the research lines reported by the members of SNOLA shows a multifaceted and intense activity, covering a wide range of topics and approaches and including connections with other LA research groups and organizations at an international level.
Most of the members of the network have a technical background. As a consequence, their research lines are mostly focused on technological contributions, as described in this paper. In spite of this bias, the members of SNOLA are aware of the global challenges in the field, are not restricted to technological issues. Among these challenges, the participants highlight the need for an ethical commitment with data; the design of systems that are able to adapt to the needs and demands of their users; and the establishment of a wider institutional framework to support and foster advances in the field. In order to face these challenges, the network might benefit from establishing connections with networks, groups, or individuals coming from complementary fields, such as pedagogy, psychology and/or philosophy. These problems are aligned with the ones pointed out by Pelánek (2020) in a recent review of the challenges of the field at an international level.
This work is just a first step towards the analysis of the impact of SNOLA in the development of LA in our context, which may be complemented with new analyses (i.e., a study on the structure of dynamics of the network), or with a deeper discussion considering research in LA at an international level.
This research has been co-funded by the National Research Agency of the Spanish Ministry of Science, Innovation and Universities and the Structural Funds (FSE and FEDER) under project grants RED2018-102725-T, TIN2017-85179-C3-1-R, TIN2017-85179-C3-2-R, TIN2017-85179-C3-3-R and TIN2016-80172-R; by FEDER/Castille and Leon Regional Government grant VA257P18; by the Basque Government under grant number IT980-16 and by the Catalan Government under grant number 2017SGR1619. This work has been co-funded by the Madrid Regional Government, through the project e-Madrid-CM (S2018/TCS-4307), the e-Madrid-CM project is also co-financed by the Structural Funds (FSE and FEDER). D. Hernández-Leo acknowledges the support by ICREA under the ICREA Academia programme.
Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542-550. https://doi.org/https://doi.org/10.1016/j.chb.2013.05.031
Alexandron, G., Ruipérez-Valiente, J. A., Chen, Z., Muñoz-Merino, P. J., & Pritchard, D. E. (2017). Copying@ Scale: Using harvesting accounts for collecting correct answers in a MOOC. Computers & Education, 108, 96-114.
Amarasinghe, I., Hernández-Leo, D., & Jonsson, A. (2019). Data-informed design parameters for adaptive collaborative scripting in across-spaces learning situations. User Modeling and User-Adapted Interaction, 1-24.
Bote-Lorenzo, M. L., & Gómez-Sánchez, E. (2018). An Approach to Build in situ Models for the Prediction of the Decrease of Academic Engagement Indicators in Massive Open Online Courses. J. UCS, 24(8), 1052-1071.
Caeiro-Rodríguez, M., Hernández-García, Á., & Muñoz-Merino, P. J. (2019). LASI-SPAIN 2019 - Conference Proceedings. Retrieved from http://ceur-ws.org/Vol-2415/
Chaparro-Peláez, J., Iglesias-Pradas, S., Rodríguez-Sedano, F. J., & Acquila-Natale, E. (2019). Extraction, Processing and Visualization of Peer Assessment Data in Moodle. Applied Sciences, 10(1). https://doi.org/10.3390/app10010163
Claros, I., Cobos, R., & Collazos, C. A. (2015). An Approach Based on Social Network Analysis Applied to a Collaborative Learning Experience. IEEE Transactions on Learning Technologies, 9(2), 190-195. https://doi.org/10.1109/TLT.2015.2453979
Cobos, R., Gil, S., Lareo, Á., & Vargas, F. (2016). Open-DLAs: An open dashboard for learning analytics. In L@S 2016 - Proceedings of the 3rd 2016 ACM Conference on Learning at Scale. https://doi.org/10.1145/2876034.2893430
Cobos, R., Jurado, F., & Blázquez-Herranz, A. (2019). A Content Analysis System that supports Sentiment Analysis for Subjectivity and Polarity detection in Online Courses. IEEE Revista Iberoamericana de Tecnologías Del Aprendizaje, 14(4), 177-187. https://doi.org/10.1109/rita.2019.2952298
Cobos, R., & Olmos, L. (2018). A Learning Analytics Tool for Predictive Modeling of Dropout and Certificate Acquisition on MOOCs for Professional Learning. In IEEE International Conference on Industrial Engineering and Engineering Management (Vol. 2018-Decem, pp. 1533-1537). IEEE. https://doi.org/10.1109/IEEM.2018.8607541
Conde, M. A., Colomo-Palacios, R., García-Peñalvo, F. J., & Larrucea, X. (2018). Teamwork assessment in the educational web of data: A learning analytics approach towards ISO 10018. Telematics and Informatics, 35(3), 551-563. https://doi.org/https://doi.org/10.1016/j.tele.2017.02.001
Conde-González, M. Á., & Hernández-García, Á. (2013). A Promised Land for Educational Decision-making?: Present and Future of Learning Analytics. In ACM International Conference Proceeding Series (pp. 239-243). https://doi.org/10.1145/2536536.2536573
Conde-González, M. Á., & Hernández-García, Á. (2019). Learning Analytics: The End of the Beginning. In Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 248-252). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3362789.3362943
de Arriba-Pérez, F., Caeiro-Rodríguez, M., & Santos-Gago, J. M. (2018). How do you sleep? Using off the shelf wrist wearables to estimate sleep quality, sleepiness level, chronotype and sleep regularity indicators. Journal of Ambient Intelligence and Humanized Computing, 9(4), 897-917. https://doi.org/10.1007/s12652-017-0477-5
Drachsler, H., & Greller, W. (2016). Privacy and Analytics – it’s a DELICATE issue. A Checklist to establish trusted Learning Analytics. 6th Learning Analytics and Knowledge Conference 2016, April 25-29, 2016, Edinburgh, UK.
EC (2016). Learning Analytics: Key messages. Retrieved from https://ec.europa.eu/education/sites/education/files/2016-pla-learning-analytics_en.pdf
EU (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC. Off. J. Eur. Union, L119 (2016), pp. 1-88.
Er, E., Dimitriadis, Y., & Gaseviç, D. (2019). An analytics-driven model of dialogic peer feedback. In 13th International Conference on Computer Supported Collaborative Learning (CSCL 2019). Lyon, France.
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., … Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy.
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71.
Gómez-Aguilar, D. A., García-Peñalvo, F. J., & Therón, R. (2014). Analítica visual en e-learning. Profesional de La Información, 23(3), 236-245. https://doi.org/10.3145/epi.2014.may.03
Guerrero-Higueras, Á. M., DeCastro-García, N., Rodriguez-Lera, F. J., Matellán, V., & Conde, M. Á. (2019). Predicting academic success through students’ interaction with Version Control Systems. Open Computer Science, 9(1), 243-251.
Hernández-García, Á., Acquila-Natale, E., Chaparro-Peláez, J., & Conde, M. Á. (2018). Predicting teamwork group assessment using log data-based learning analytics. Computers in Human Behavior, 89, 373-384. https://doi.org/https://doi.org/10.1016/j.chb.2018.07.016
Hernández-García, Á., & Suárez-Navas, I. (2017). GraphFES: A Web Service and Application for Moodle Message Board Social Graph Extraction. In B. Kei Daniel (Ed.), Big Data and Learning Analytics in Higher Education: Current Theory and Practice (167-194). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-06520-5_11
LACE (2020). Learning Analytics Community Exchange Project Webpage, http://www.laceproject.eu
Latour, B. (2005). Reassembling the social. An introduction to actor-network-theory. Oxford: Oxford University Press.
Manso-Vázquez, M., Caeiro-Rodríguez, M., & Llamas-Nistal, M. (2018). An xAPI Application Profile to Monitor Self-Regulated Learning Strategies. IEEE Access, 6, 42467-42481.
Martínez, J. A., Campuzano, J., Sancho-Vinuesa, T., & Valderrama, E. (2019). Predicting student performance over time. A case study for a blended-learning engineering course. CEUR Workshop Proceedings, 2415, 43–55. Retrieved from http://ceur-ws.org/Vol-2415/paper05.pdf
Menchaca Sierra, I., Guenaga, M., & Solabarrieta, J. (2018). Learning analytics for formative assessment in engineering education. The International Journal of Engineering Education, 34(3), 953-967.
Michos, K., Hernández-Leo, D., & Albó, L. (2018). Teacher-led inquiry in technology-supported school communities. British Journal of Educational Technology, 49(6), 1077-1095.
Moreno-Marcos, P. M., Muñoz-Merino, P. J., Maldonado-Mahauad, J., Pérez-Sanagustín, M., Alario-Hoyos, C., & Delgado Kloos, C. D. (2020). Temporal analysis for dropout prediction using self-regulated learning strategies in self-paced MOOCs. Computers & Education, 145, 103728.
Muñoz-Merino, P. J., Novillo, R. G., & Delgado Kloos, C. D. (2018). Assessment of skills and adaptive learning for parametric exercises combining knowledge spaces and item response theory. Applied Soft Computing, 68, 110-124.
Omedes, J. (2018). Learning Analytics 2018 - An updated perspective. Retrieved from https://www.iadlearning.com/learning-analytics-2018.
Papamitsiou, Z., Giannakos, M., and Ochoa, X. (2020). From childhood to maturity: Are we there yet?: Mapping the intellectual progress in learning analytics during the past decade, in Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 559-568). https://doi.org/10.1145/3375462.3375519
Pelánek, R (2020). Learning Analytics Challenges: Trade-offs, Methodology, Scalability, In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, (pp. 554-558). https://doi.org/10.1145/3375462.3375463
Peña-Ayala, A. (2018). Learning analytics: A glance of evolution, status, and trends according to a proposed taxonomy. WIREs Data Mining and Knowledge Discovery, 8(3), e1243. https://doi.org/10.1002/widm.1243
Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330-343.
Ros, S., Lázaro, J. C., Robles-Gómez, A., Caminero, A. C., Tobarra, L., & Pastor, R. (2017). Analyzing Content Structure and Moodle Milestone to Classify Student Learning Behavior in a Basic Desktop Tools Course. In Proc. of Intl. Conference Technological Ecosystems for Enhancing Multiculturality (TEEM). Cádiz, Spain.
Rubio-Fernández, A., Muñoz-Merino, P. J., & Delgado Kloos, C. (2019). A learning analytics tool for the support of the flipped classroom. Computer Applications in Engineering Education, 27(5), 1168-1185.
Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Leony, D., & Delgado Kloos, C. (2015). ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior, 47, 139-148.
Ruiz, S., Urretavizcaya, M., Rodríguez, C., & Fernández-Castro, I. (2018). Predicting students’ outcomes from emotional response in the classroom and attendance. Interactive Learning Environments, 1–23. https://doi.org/10.1080/10494820.2018.1528282
Slade, S., & Prinsloo, P. (2013). Learning analytics ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510-15
SOLAR (2020). Society for Learning Analytics Research webpage: http://solaresearch.org
SNOLA (2020). Spanish Network of Learning Analytics: http://www.snola.es
Tobarra, L., Ros, S., Hernández, R., Robles-Gómez, A., Caminero, A. C., & Pastor, R. (2014). Integrated Analytic dashboard for virtual evaluation laboratories and collaborative forums. In 2014 XI Tecnologías Aplicadas a la Enseñanza de la Electrónica (Technologies Applied to Electronics Teaching) (TAEE) (pp. 1-6).
Tobarra, L., Ros, S., Hernández, R., Robles-Gómez, A., Pastor, R., Caminero, A. C., … Claramonte, J. (2017). Analyzing Students’ Behavior in UNED-COMA MOOCs. In LASI-SPAIN (pp. 124–137).
Tsai, Y.-S., Gašević, D., Whitelock-Wainwright, A., Muñoz-Merino, P. J., Moreno-Marcos, P. M., Fernández, A. R., … others. (2018). SHEILA: Support Higher Education to Integrate Learning Analytics.
Vázquez-Ingelmo, A., García-Peñalvo, F. J., & Therón, R. (2019). Taking advantage of the software product line paradigm to generate customized user interfaces for decision-making processes: A case study on university employability. PeerJ Computer Science, 5. https://doi.org/10.7717/peerj-cs.203
Villamañe, M., Alvarez, A., & Larrañaga, M. (2020). CASA, An Architecture to Support Complex Assessment Scenarios. IEEE Access. https://doi.org/10.1109/ACCESS.2020.2966595
Villamañe, M., Larrañaga, M., & Álvarez, A. (2017). Rating monitoring as a means to mitigate rater effects and controversial evaluations. In Proceedings of the 5th International Conference on Technological Ecosystems for Enhancing Multiculturality - TEEM 2017 (pp. 1-8). Cádiz, Spain: ACM Press. https://doi.org/10.1145/3144826.3145389
Vujovic, M., & Hernández-Leo, D. (2019). Shall We Learn Together in Loud Spaces? Towards Understanding the Effects of Sound in Collaborative Learning, 891-892.
Wiley, K., Dimitriadis, Y., Bradford, M., & Linn, M. (2020). From Theory to Action: Developing and Evaluating Learning Analytics for Learning Design. In Learning Analytics and Knowledge Conference (LAK 2020). Frankfurt, Germany
Notas de autor