Estudios e investigaciones / Research and Case Studies
Scale for self-evaluation of competencies in online teaching: A systematic study of design and validation
Escala de autoevaluación de competencias docentes en enseñanza virtual: Un estudio sistemático de diseño y validación
Scale for self-evaluation of competencies in online teaching: A systematic study of design and validation
RIED-Revista Iberoamericana de Educación a Distancia, vol. 28, núm. 1, 2025
Asociación Iberoamericana de Educación Superior a Distancia

How to cite: Reyes-Vásquez, M. N. (2025). Scale for self-evaluation of competencies in online teaching: A systematic study of design and validation. [Escala de autoevaluación de competencias docentes
en enseñanza virtual: Un estudio sistemático de diseño y validación]. RIED-Revista Iberoamericana de Educación a Distancia, 28(1). https://doi.org/10.5944/ried.28.1.41320
Abstract: The Honduran Higher Education System lacks instruments that evaluate various constructs associated with the online teaching modality, which constitute as evaluation models aimed to monitor and assure the quality of higher education in its online teaching modality. Hence, this study was oriented to design an instrument that would gather the required psychometric properties for the valid and reliable measurement of the competencies in online teaching of university professors, during the year 2023. For this purpose, a mixed research approach was used, but with a greater quantitative weight. The sample was non-probabilistic, by convenience and was integrated by 650 university professors from the UNAH, UPNFM and EAP Zamorano. The Scale for self-evaluation of competencies in online teaching (ECODEV) was constructed through a systematic review of the literature under three theoretical factors: pedagogical, technological and course management competences. The findings ratified that the construct under measurement is composed of these factors with a total of 49 items. The factorial model of the ECODEV corresponds to a hierarchical model and it is explained by a total cumulative variance of 0.66, a total Ω of 0.99 and with fit indexes CFI = 0.999, RMSEA = 0.043 and SRMR = 0.045. In conclusion, the ECODEV has a satisfactory psychometric quality and, therefore, provides robust and relevant information about its measurable construct in university teachers in the Honduran context.
Keywords: evaluation, digital competencies, university teachers, content validity, construct validity, reliability.
Resumen: El Sistema de Educación Superior de Honduras carece de instrumentos que evalúen diversos constructos asociados a la modalidad a distancia en su expresión virtual los cuales se constituyan como modelos de evaluación encaminados al monitoreo y aseguramiento de la calidad en esta modalidad. Este estudio estuvo orientado a diseñar un instrumento que reuniera las propiedades psicométricas necesarias para la medición válida y confiable de las competencias en enseñanza virtual del profesorado universitario, durante el año 2023. Para este fin se empleó un enfoque de investigación mixto, con un mayor peso cuantitativo. La muestra fue de tipo no probabilística, por conveniencia y estuvo compuesta por 650 profesores universitarios de la UNAH, UPNFM y EAP Zamorano. La Escala de autoevaluación de competencias docentes en enseñanza virtual (ECODEV) fue construida mediante revisión sistemática de la literatura bajo tres factores teóricos: competencias pedagógicas, tecnológicas y de gestión y manejo del curso. Los hallazgos ratificaron que el constructo en medición se compone de estos factores con un total de 49 ítems. El modelo factorial de la ECODEV corresponde a un modelo jerárquico y es explicado por una varianza total acumulada de 0.66, un Ω total de 0.99 y con índices de ajuste CFI = 0.999, RMSEA = 0.043 y SRMR = 0.045. En conclusión, la ECODEV posee una calidad psicométrica satisfactoria por lo que aporta una robusta y pertinente información sobre su constructo medible en profesores universitarios en el contexto hondureño.
Palabras clave: evaluación, competencias digitales, profesores universitarios, validez de contenido, validez de constructo, confiabilidad.
INTRODUCTION
The unexpected migration from the traditional face-to-face modality to the distance modality in its online expression as a result of the global health emergency caused by COVID- 19 revealed the limited development that this modality had in the country prior to this event. According to UNESCO (2017), in Latin America and the Caribbean, the online modality dates after 1995 and was notably strengthened after the end of that decade. In the particular case of Honduras, Quintanilla (2016) points out that the genesis of this modality occurred until 2005. Despite this, in this region the adoption and development of virtuality in higher education is incipient and relatively low. This is due to certain factors such as the fact that “the region’s IT and telematics infrastructure is far from being at the level of advanced countries where online higher education has become more widespread” (UNESCO, 2017, pp. 13 – 14). In addition, Nicola et al. (2020) state that the limited regulations and resources in the region represent factors that make efficiency in education impossible when facing situations such as the one caused by the COVID-19 pandemic (as cited in Cadena & Ramos, 2023).
Prior to the rise of this phenomenon, the development of online education in the region, and especially in Honduras, was still meager due to the social inequality gaps that prevail in the country. In 2023, only half of the population had access to the Internet, equivalent to a rate of 53.4 %, but by 2019 (before the apogee of the pandemic) the population access rate was only 39.4 %. Moreover, only 16.5 % accessed this service from a computer (Instituto Nacional de Estadística, 2019, 2023). Hence, this health phenomenon evidenced multiple areas for improvement in distance education in its online expression not only in the national but also in the regional context. One of the most important areas lies in the competencies evaluation of the teachers who facilitate teaching in online environments ( International Commission on the Futures of Education, 2020; Meinck et al., 2022; Organization for Economic Cooperation and Development, 2021).
The expansion of the online modality in higher education is an imminent reality in the post-pandemic era (Carbonell et al., 2021). Therefore, in such a scenario, several needs arise within the framework of quality assurance of higher education in its distance modality in its online expression in the country. Needs such as the formulation of a model for the evaluation of teaching competencies in online education that provides the necessary elements for the issuance of technical and/or professional judgments based on a valid and reliable instrument according to the national context. Therefore, the present study was oriented to design a scale that would gather the psychometric properties necessary for the valid and reliable measurement of the competencies in online teaching possessed by the university professors of the HEIs of Honduras, during the academic year 2023.
METHODOLOGY
Design and sample
In order to achieve the objective of this study, a mixed research approach was used, but with a greater quantitative weight due to the psychometric nature of the research, which focused on the measurement of the latent variable “competencies in online teaching”. It was of cross-sectional temporality and non-experimental design. The sample of this study was non-probabilistic, by convenience and it was made up of 650 university professors. 66.6 % belonged to the Universidad Nacional Autónoma de Honduras (UNAH), 31.5 % to the Universidad Pedagógica Nacional Francisco Morazán (UPNFM) and 1.8 % to the Escuela Agrícola Panamericana Zamorano (EAP Zamorano). The sample was balanced in terms of the sex of the participants. 51.8 % were men and 48.2 % were women. This sample is comparable to the actual distribution of the university professors, 52.7 % of whom were men and 47.3 % women (UNAH, 2021; UPNFM, 2022). The predominant age range was 41 – 45 years (16.5 %). Most of the sample had postgraduate studies at the master’s degree level (70.9 %). 31.5 % of university professors had three years of experience teaching courses in online environments and 35.6 % had more than 3 years of experience in this modality.
Instrument
The Scale for self-evaluation of competencies in online teaching (known in Spanish by the acronym ECODEV) was constructed in the framework of this study through a systematic review of the literature considering the theoretical contributions on online teaching competencies of Albrahim (2020) and Farmer and Ramsdale (2016). Furthermore, due to the imminent need to generate an input according to and contextualized to the national reality, the contributions of the Educational Model of the UNAH (2009), the Regulations for Distance Education at the Higher Education Level in Honduras (2014) and the Conceptual Document on Distance Education at the UNAH (2014) were also considered. The thematic lines or theoretical dimensions for the design of the item bank of the scale were three. The first refers to pedagogical competencies, which are the abilities possessed by the teacher who facilitates the learning process in online environments to ensure that the course curriculum and the students’ learning experiences support the expected outcomes of the course. It also includes components such as the application of didactic strategies, learning activities and assessments that promote active student participation and interaction (Albrahim, 2020; DES, 2014; Farmer & Ramsdale, 2016; UNAH, 2009, 2014).
The second refers to technological competencies which precisely addresses the teacher’s abilities to select, organize and manage technology for learning in the course, integrating not only diversity but also adequate technological tools and resources that are aligned with the different learning styles, nature of the courses and the students’ abilities to use them so as to ensure that the teaching-learning process is appropriate, productive and meaningful (Albrahim, 2020; DES, 2014; Farmer & Ramsdale, 2016; UNAH, 2009, 2014). Finally, the third dimension refers to course management competencies. This addresses abilities to establish a positive and assertive learning environment in the online setting. The teacher must be able to foster a genuine supportive relationship with students that contributes to their academic, personal and professional growth and, moreover, the teacher must be able to foster an inclusive learning community by facilitating opportunities for students to actively interact, discuss and work collaboratively (Albrahim, 2020; DES, 2014; Farmer & Ramsdale, 2016; UNAH, 2009, 2014).
The ECODEV is a self-report and polytomous instrument. It is aimed to measure, from a self-critical perspective, the level of mastery of competencies in online teaching possessed by university professors who apply or have applied instructional practices in online environments. A bank of 68 items was submitted for content validation. The pilot scale submitted for construct validation was constituted by 50 items segmented into three dimensions: pedagogical competencies (18 items), technological competencies (12 items), and course management competencies (20 items). Each item is scored on a 4-point numerical scale to self-evaluate the level of competency mastery of each item (1 = I need to develop the competency; 2 = I have an emerging level of development of the competency; 3 = I have an intermediate level of development of the competency; 4 = I have a high level of development of the competency). It was empirically hypothesized that the theoretical factor structure of the ECODEV would be a hierarchical or second-order model. This hypothesized model guided the systematic process to construct the scale.
Procedure
The ECODEV was applied digitally through Microsoft Forms and was disseminated to the institutional e-mails of the university professors of the three stated universities. Regarding the UNAH, it was mainly disseminated through the Departamento de Investigación de Opinión Pública of the Dirección de Comunicación Estratégica of this HEI. The Department Heads of the different faculties and university campuses and the Instituto de Profesionalización y Superación Docente also supported to do so. At the UPNFM, it was disseminated through the teachers’ immediate authorities (Department Heads) by instructions from the Dean’s Office, the Academic Vice-Rectory, the Vice-Rectory of the University Distance Education Center (CUED) and the Technical Assistant of the Vice-Rectory of Research and Graduate Studies. As for the EAP Zamorano, the scale was disseminated through the Department of Teaching Development and Educational Quality of the Associate Dean’s Office of Academic Quality Management. The application period was from March 24th to August 11th, 2023.
Data analysis
The content validity of the scale was determined using the content validity index (CVI) technique (Yusoff, 2019). The instrument was subjected to validation through a panel of experts that consisted of seven professionals with academic formation and professional experience in areas such as online education, educational computing, instructional technology, innovation and educational evaluation. The lower limit of acceptable value set for both the item-level content validity index (I-CVI) and scale-level content validity index based on the average method (S-CVI/Avg) was 0.80 (Polit et al., 2007, as cited in Yusoff, 2019). To analyze the construct validity of the ECODEV, an Exploratory Factor Analysis (EFA) and a Confirmatory Factor Analysis (CFA) were applied using the database downloaded directly from Microsoft Forms and performed using R Studio version 2022.07.1.
The sample was divided into two equal parts to run the factor analyses (Abad et al., 2011; Izquierdo et al., 2014; Muñiz & Fonseca-Pedrero, 2019). Both samples were composed of 325 assessments so the item – evaluated ratio was approximately 6 to 1 or, from another perspective, both samples exceeded the required plausible sample number proposed by several authors (n = 200) (Abad et al., 2011; Izquierdo et al., 2014; Muñiz & Fonseca-Pedrero, 2019; Rigo & Donolo, 2019). To perform the EFA, a polychoric correlation matrix was used as a starting point, in accordance with the polytomous nature of the instrument and, consequently, the ordinal nature of the data. Bartlett’s Test of Sphericity and the Kaiser-Meyer-Olkin Test were applied to determine whether the correlation matrix extracted was factorizable and whether it revealed a factor structure.
The acceptance criteria were ≥ 0.35 for total inter-item correlations, a significance value < 0.05 for Barlett’s Test of Sphericity and a value ≥ 0.70 for the Kaiser-Meyer-Olkin Test (Izquierdo et al., 2014). Parallel analysis was applied as the retention method due to its fit properties with categorical variables using polychoric correlation matrices (Izquierdo et al., 2014). The extraction method used was Principal Axis Factorization due to its use does not depend on normality assumptions and its robustness allows the treatment of unequal factor loadings, as well as the recovery of weak factors. The factor rotation method applied was the Promax Solution due to its oblique nature whose use lends itself to both hierarchical and non- hierarchical data (Grieder & Steiner, 2021). Having obtained the factor loadings of the items per factor through the preceding methods, a cutoff point of > 0.40 was applied adhering to this strict criterion as suggested by MacCallum et al. (1999), Velicer and Fava (1998) and Williams et al. (2010) when the sample for the AFE is less than 300 cases to obtain the solution and final distribution of items per factor (as cited in Lloret-Segura et al., 2014).
The CFA was performed to examine the goodness of fit of the model suggested by the EFA. The hierarchical model that guided the construction of the scale and a unidimensional model were also subjected to analysis. Due to the ordinal nature of the data, the Diagonally Weighted Least Squares (DWLS) estimator was used (Ferrando et al., 2022; Rigo & Donolo, 2019). Model fit was evaluated by considering both inferential and descriptive goodness-of-fit indices such as χ2, CFI, TLI, RMSEA and the SRMR. For this, the acceptance criteria proposed by Abad et al. (2011) were taken as a reference. They point out that the model fits if the p-value ≥ 0.05 for the χ2, values ≥ 0.95 for the CFI and TLI indices, a value ≤ 0.06 for the RMSEA and for the SRMR index, a value ≤ 0.08. Finally, to determine the reliability of the ECODEV, the internal consistency of the scale and its three factors was examined using the McDonald Omega Coefficient due to its greater precision when considering the proportion of the true variance in the observed scores (Flora, 2020; Ventura-León & Caycho-Rodríguez, 2017).
RESULTS
Content validity of the ECODEV
The preliminary version of the item bank (68) achieved an acceptable CVI, but 8 of them did not reach the cut-off value at the item level. These were discarded and the remaining set (60) were retested and obtained higher values. However, one of the frequent qualitative judgments among the experts was the suggestion to eliminate or unify some items whose essence tended to indicate similarity. Consequently, 6 items were eliminated, and 8 items were unified and, thus, the set of items was reduced to 50. The results suggest that the content of the items in the final version of the ECODEV evidences excellent and satisfactory validity (Polit et al., 2007, as cited in Yusoff, 2019), as shown in Table 1. The results and comparisons between the three versions are detailed in the same table.
| Scale dimensions | Content validity | ||
|
S-CVI/Avg (68 items) |
S-CVI/Avg (60 items) |
S-CVI/Avg (50 items) |
|
| Pedagogical competencies | 0.90 | 0.93 | 0.94 |
| Technological competencies | 0.90 | 0.96 | 0.98 |
| Course management competencies | 0.94 | 0.95 | 0.97 |
| Global scale calculation | 0.93 | 0.96 | 0.98 |
Construct validity of the ECODEV
Exploratory Factor Analysis
The inter-item correlations were satisfactory overall, and these ranged from 0.434 to 0.962. Barlett’s Test of Sphericity indicated a chi-square of 18725.42 (gl = 1225), with a p-value of 0 (< 0.05) and the KMO Test indicated a value of 0.77. Parallel analysis suggested the retention of 3 factors which was supported by the optimal coordinates method, as shown in Figure 1. The percentage of variance explained based on eigenvalues for the three factors represents 78.55 %; the first factor explains about 69.45 % of the variance, the second 5.65 % and the third 3.45 %.

When applying the extraction and rotation method, all the items loaded on the factor for which they had been designed theoretically and empirically, except for only two. However, the dimensions changed their order of construct conformation. The results suggest that the first factor that comprises the construct “competencies in online teaching” is the dimension Course Management competencies dimension, the second is the one referring to Pedagogical competencies, and the third, Technological competencies. Table 2 presents the standardized factor loadings of the scale items for the three different factors according to the AFE.
| Item | # Item | Factor 1 | Factor 2 | Factor 3 |
| I demonstrate sensitivity and empathy in resolving conflicts and misunderstandings in an amicable manner. | G44 | 0.96 | -0.02 | -0.05 |
| I demonstrate respect, patience, and responsiveness to students in all communications. | G42 | 0.95 | -0.01 | -0.03 |
| I show interest and concern so that students are learning. | G40 | 0.93 | 0.03 | -0.03 |
| I foster an environment of respect and equity in the course. | G41 | 0.92 | 0.00 | -0.01 |
| I offer advice, suggestions and clarify doubts in a timely manner. | G50 | 0.90 | 0.02 | -0.01 |
| I help students resolve conflicts through consensus and mutual understanding by fostering communications skills. | G45 | 0.90 | 0.01 | -0.02 |
| I offer advice and information (e.g., about the subject, learning process) as requested by students. | G46 | 0.85 | 0.08 | -0.03 |
| I maintain a positive and constructive attitude in the face of change, adversity and stressful situations. | G43 | 0.84 | -0.05 | 0.08 |
| I use different methods of communication to ensure accessibility with my students, and students with their classmates. | G48 | 0.83 | -0.06 | 0.11 |
| I promote assertive individual communication by responding to messages in a reasonable time frame. | G49 | 0.82 | 0.01 | 0.07 |
| I demonstrate leadership, management, mentoring and training skills during the course. | G39 | 0.70 | 0.01 | 0.18 |
| I provide clear instructions to keep course participants focused on learning tasks and activities. | G37 | 0.67 | 0.10 | 0.10 |
| I encourage excellence in students’ work through motivation and facilitation processes. | G36 | 0.59 | 0.18 | 0.11 |
| I encourage students to collaborate through team tasks, projects and discussions. | G32 | 0.59 | 0.30 | -0.12 |
| I create a safe and supportive online learning environment for didactic dialogue through effective online communication, interaction and course management. | G38 | 0.59 | 0.02 | 0.26 |
| I refer students to appropriate sources of support (academic services, advising, tutoring) when needed. | G47 | 0.58 | 0.10 | 0.12 |
| I recognize the importance of the contributions of colleagues and students to my success. | G31 | 0.58 | 0.11 | 0.16 |
| I facilitate spaces for interactive discussion and exchange of ideas in a synchronous and an asynchronous manner. | G34 | 0.44 | 0.24 | 0.11 |
| I promote peer-to-peer learning to produce a meaningful exchange of ideas and learning. | G33 | 0.37 | 0.36 | -0.05 |
| I develop learning activities that allow students to construct knowledge and develop skills and/or competencies. | P4 | 0.00 | 0.92 | -0.12 |
| I link learning activities to the outcomes I expect from the course. | P5 | 0.04 | 0.87 | -0.10 |
| I adapt the didactic strategies based on the results obtained before, during and after the online evaluation throughout the course. | P18 | 0.02 | 0.79 | 0.00 |
| I use learning strategies that provide opportunities for authentic practice of what has been learned. | P9 | 0.00 | 0.78 | 0.05 |
| I encourage students’ self-assessment and reflection on their learning process. | P14 | -0.06 | 0.78 | -0.03 |
| I implement didactic strategies appropriate to the nature of the course. | P7 | 0.03 | 0.78 | 0.01 |
| I consider learning styles when creating or selecting the course learning activities. | P2 | 0.01 | 0.76 | -0.01 |
| I integrate didactic strategies that promote student-centered learning. | P1 | -0.09 | 0.75 | 0.15 |
| I use complementary methods for the assessment of students’ learning. | P12 | 0.07 | 0.74 | -0.04 |
| I create opportunities to assess students’ prior knowledge when introducing new content. | P3 | 0.09 | 0.73 | -0.06 |
| I establish learning activities consistent with the students’ technological accessibility. | P6 | 0.03 | 0.72 | 0.07 |
| I apply a diagnostic, formative and summative assessment that allows me to monitor students’ learning adequately. | P15 | -0.09 | 0.71 | 0.03 |
| I create assessment activities aligned with the outcomes I expect from the teaching-learning process. | P13 | -0.01 | 0.71 | 0.12 |
| I use feedback strategies in identifying strengths and areas of opportunity in the students’ learning. | P17 | -0.01 | 0.68 | 0.16 |
| I make sure that the didactic strategies applied in the course are adjusted with the technological tools. | P8 | -0.01 | 0.66 | 0.20 |
| I design learning material relevant to the achievement of the course objectives. | P11 | 0.02 | 0.63 | 0.13 |
| I develop study guides with online access to optimize the students’ learning experience. | P10 | -0.07 | 0.43 | 0.32 |
| I evaluate which technological tools are most effective in achieving the results I expect from the course. | T29 | -0.05 | 0.02 | 0.88 |
| I effectively manage the technological tools I use for teaching in online modality. | T21 | 0.03 | -0.06 | 0.83 |
| I provide support to students for the correct use of the technological tools of the course. | T22 | -0.04 | 0.09 | 0.78 |
| I create digital content such as digital materials and instructional videos for the course. | T26 | -0.06 | 0.00 | 0.78 |
| I evaluate the effectiveness of online resources and materials through reflection on the online teaching experiences to monitor and improve my teaching. | T30 | -0.01 | 0.06 | 0.77 |
| I facilitate teaching spaces synchronously and asynchronously. | T20 | 0.05 | 0.01 | 0.72 |
| I provide feedback to students within a reasonable response time using basic technological communication tools (platform, messages, comments, e-mail). | T28 | 0.06 | 0.07 | 0.71 |
| I use open access resources and technology tools in the online learning environment to create meaning and relevance in the students’ learning. | T19 | 0.22 | -0.07 | 0.71 |
| I select digital educational resources that promote motivation and active participation of students. | T27 | 0.00 | 0.15 | 0.66 |
| I implement appropriate strategies to manage students’ workload. | T23 | 0.05 | 0.23 | 0.62 |
| I include resources from a variety of sources (e.g., textbooks, articles, Internet, personal experiences, guest speakers/experts, students’ experience and knowledge). | T25 | 0.21 | 0.07 | 0.59 |
| I update learning resources for their distribution to the students in the course. | T24 | 0.10 | 0.21 | 0.57 |
| I use course statistics to follow up and monitor students’ learning process. | P16 | -0.17 | 0.37 | 0.51 |
| I motivate students by providing them with authentic learning opportunities in the online environment. | G35 | 0.39 | 0.03 | 0.45 |
After identifying the items with satisfactory factor loadings, the following distribution of items was obtained. Factor 1 “Course management competencies” is made up of 18 items. All of them had been theoretically designed for this factor. Item G33, which was originally part of this subscale, was deleted because it showed double factor loadings, one with its original theoretical factor and one with Factor 2 “Pedagogical competencies”. None of them exceeded the cut-off point (see Table 2). Factor 2 “Pedagogical competencies” is made up of 17 items. Like the previous factor, all of them had been theoretically designed for this factor and only one of them showed behavior far from what was expected (item P16, see Table 2).
Finally, Factor 3 “Technological competencies” is made up of 14 items. All the items designed for this factor loaded satisfactorily (those with initial nomenclature T); however, two items from the other factors showed satisfactory factor loadings for this factor (items P16 and G35). Although both items did not fit in the factor for which they had been theoretically designed, they were not eliminated as suggested by the literature (Izquierdo et al. 2014; Muñiz & Fonseca-Pedrero, 2019) due to their identification and significance for the factor “Technological competencies” (see Table 2). The cumulative variance explained by this three- factor model is 0.66. Table 3 summarizes the final distribution of items with satisfactory factor loadings per factor.
| Factor 1: Course Management C. | Factor 2: Pedagogical C. | Factor 3: Technological C. | |||
| Item | Factorial loading | Item | Factorial loading | Item | Factorial loading |
| G44 | 0.958 | P12 | 0.739 | T29 | 0.88 |
| G42 | 0.954 | P4 | 0.921 | T21 | 0.829 |
| G40 | 0.927 | P5 | 0.865 | T22 | 0.785 |
| G41 | 0.924 | P9 | 0.784 | T26 | 0.781 |
| G50 | 0.902 | P13 | 0.708 | T30 | 0.772 |
| G45 | 0.889 | P3 | 0.733 | T20 | 0.721 |
| G46 | 0.846 | P11 | 0.63 | T28 | 0.709 |
| G43 | 0.835 | P2 | 0.764 | T19 | 0.707 |
| G48 | 0.828 | P10 | 0.431 | T27 | 0.658 |
| G49 | 0.818 | P6 | 0.718 | T23 | 0.618 |
| G39 | 0.703 | P8 | 0.657 | T25 | 0.59 |
| G37 | 0.674 | P7 | 0.778 | T24 | 0.57 |
| G36 | 0.595 | P1 | 0.748 | P16 | 0.515 |
| G32 | 0.591 | P14 | 0.779 | G35 | 0.452 |
| G38 | 0.589 | P18 | 0.791 | ||
| G47 | 0.584 | P15 | 0.712 | ||
| G31 | 0.577 | P17 | 0.676 | ||
| G34 | 0.444 | ||||
As a result, the ECODEV is composed of 49 items segmented into three subscales which share a high correlation as illustrated in Table 4.
| Factor 1 | Factor 2 | Factor 3 | |
| Factor 1 | 1.00 | 0.73 | 0.76 |
| Factor 2 | 0.73 | 1.00 | 0.79 |
| Factor 3 | 0.76 | 0.79 | 1.00 |
Confirmatory Factor Analysis
The results showed that both the three-factor correlated model, and the hierarchical model met the acceptance criteria in relation to the descriptive and inferential fit indices. However, both models revealed invariant values, as shown in Table 5. The unidimensional model also met the acceptance criteria, but for this model, the CFI and TLI fit indices were subtly below those obtained by its predecessors. The SRMR and RMSEA fit indices for this model exceeded the values proposed by Abad et al. (2011): 0.082 > 0.08 and 0.111 > 0.06 correspondingly, suggesting, therefore, that the model does not fit. Table 5 shows the fit indices of the examined models.
| Model | χ2 | gl | p | CFI | TLI | RMSEA (CI) | SRMR |
| Three correlated factors | 1797.090 | 1124 | .000 | 0.999 | 0.999 | 0.043 (0.039 – 0.047) | 0.045 |
| Hierarchical model | 1797.090 | 1124 | .000 | 0.999 | 0.999 | 0.043 (0.039 – 0.047) | 0.045 |
| One-dimensional model | 5626.601 | 1127 | .000 | 0.993 | 0.993 | 0.111 (0.108 – 0.114) | 0.082 |
These findings suggest that the construct “competencies in online teaching” has a general factor that explains both the factors of the construct as well as their items. The phenomenon that the first two models have equal fit indices suggests that the addition of a general factor to the construct does not alter the factor structure initially proposed by the AFE, but only adds an evidence of construct validity that was not considered by this one. Therefore, it is concluded that the ECODEV has a hierarchical or second-order factor model as illustrated in Figure 2.

Reliability of the ECODEV
The findings suggest a satisfactory internal consistency for the general scale and its subscales (Flora, 2020; Ventura-León & Caycho-Rodríguez, 2017). The first subscale “Course management competencies” obtained Ω of 0.97, the subscale “Pedagogical competencies” reached a Ω of 0.96, and the subscale “Technological competencies” obtained a Ω of 0.95. The overall scale reached a highly satisfactory value of Ω = 0.99.
DISCUSSION
The ECODEV was constructed in order to respond to the objective of this study. Its content and construct validation contributed to avoid the overrepresentation of the variables in the theoretically supported dimensions (Muñiz & Fonseca-Pedrero, 2019). The factor analyses ratified the existence of the three factors that had been theoretically supported through systematic review of the literature, which supports that the process that conducted the design of the pilot scale was systemic, rigorous and satisfactory (Ferrando et al., 2022; Izquierdo et al., 2014; Lloret-Segura, 2014; Muñiz & Fonseca-Pedrero, 2019). Furthermore, these findings showed that the three factors of the scale are highly positively correlated. That is to say that the course management, pedagogical and technological competencies have a direct relationship and interaction with each other.
The variance accumulated by this model suggests an adequate factorial model (Abad et al. 2011; Mullo & Marcatoma, 2022) and, in addition, it shows an equitable distribution in terms of the observable variables which statistically ensures a measurement without under- or over-representation of the dimensions of the construct under study (Muñiz & Fonseca-Pedrero, 2019). The subscale Course Management competencies obtained the best results in relation to the items factor loadings of the entire scale suggesting an excellent and adequate fit to the measured factor.
In addition, this subscale had been originally designed and positioned theoretically as the third dimension of the construct “competencies in online teaching”; however, through the construct validation, this dimension was positioned as the first dimension (factor) of the scale. In other words, it is the factor that provides greater psychometric information for measuring the teaching competencies in online environments in the context of higher education in Honduras. The findings also suggest that the scale has three factors that, although they are correlated, there is also a general factor above them that explains them, as well as their observable variables. This general factor is called “competencies in online teaching” and it was the latent variable of greatest interest in the study.
This evidence suggests that the ECODEV has excellent psychometric properties of construct validity which statistically guarantees that the scale efficiently measures online teaching competencies in university teachers (Izquierdo et al., 2014; Ferrando et al., 2022; Lloret-Segura et al., 2014). The scale also possesses excellent reliability properties. McDonald’s omega coefficients suggested satisfactory internal consistency, so this scale is also reliable for the task at hand (Flora, 2020; Ventura-León & Caycho-Rodríguez, 2017). Therefore, the ECODEV provides robust and relevant information on the construct “competencies in online teaching” in university professors in the Honduran context, which enables it to serve as a reliable and valid input for the issuance of solidly supported technical and professional judgments.
Nevertheless, even though the scale showed very good psychometric characteristics, several methodological limitations and deficiencies are recognized. In the first instance, despite having a broad and gender-balanced sample, this was non-probabilistic, which in statistical terms does not allow the generalization of the results to all university professors in Honduran HEIs. The validation of the ECODEV was carried out with the participation of only the two largest public universities in the country (UNAH and UPNFM) and only the EAP Zamorano participated with a significantly small sample, which does not provide the required representativeness regarding the country’s private universities.
Hence, one of the emerging lines of research is the validation of the scale in this context (private universities of the country) to check if the factorial model of the ECODEV adjusts or varies for the country’s private sector universities. Another limitation has to do with the cross-sectional nature of the research, which prevented the observation of changes in the construct over time. On the other hand, due to the nature of the instrument (self-report) and its application modality, the possible presence of subjective biases in the research results is as well recognized. In other words, there is a possibility that participants may not have exercised a critical and genuine judgment on the measured construct (Podsakoff et al., 2003, as cited in Elosua et al., 2023; Garcia-Pardina et al., 2024).
It must be recognized that this scale arose in an attempt to generate an evaluation model that contributes to the quality assurance of higher education in its online modality, so that any judgment that is not gestated from a true reflection of the teaching practice hardly contributes to this end. In this process of improving the quality of education, the teacher is the key element whose reflective practice of his or her teaching should be the central axis (Meierdirk, 2016; Salom, 2018, as cited in Rico-Reintsch, 2019). Therefore, a process of self-reflection that does not genuinely reflect the reality of educational practices will hardly facilitate the generation of spaces or models for teacher development and growth at a personal and professional level.
“What is not self-evaluated is devalued and this does not allow innovation in classroom management and in the creation of innovative, relevant and complex situations in the teaching and learning process and its different components” (Rico-Reintsch, 2019, p.70). A proposal to mitigate this possibility in future research is the validation of the ECODEV with a different rating scale that does not allude to the self-assessment of the teacher’s competence level but to the frequency with which the teacher develops the practice in online teaching reported by the item. For example: (1) Never, (2) Rarely, (3) Occasionally, (4) Frequently and (5) Very frequently.
Added to this limitation is the lack of data triangulation, which represents a reduction in the depth and complementary validity of the results. Future research should consider this fact and validate the ECODEV construct from the perspective of the students and, if possible, of the immediate authorities of the teachers. This would allow a triangulation of data with these three sources of information and, in addition, would provide greater evidence of construct validity than that obtained in this study or it would broaden the possible uses that the scale has initially established.
The ECODEV represents a first exercise aimed to the evaluation of competencies in online teaching, which should not necessarily be taken as an optimal input. The limitations discussed do not invalidate the results of this study, nor the evaluation model formulated from them, but only expose a critical position that researchers or evaluators should consider when using and making judgments based on the ECODEV. There is still work to do to examine this phenomenon in greater depth in the Honduran context and its approach will contribute to the consolidation of a genuine and psychometrically solid and robust model for measuring the construct “competencies in online teaching” in the Honduran Higher Education System.
CONCLUSIONS
The present research was a pioneering study in Honduras and in the regional context in designing and analyzing the psychometric properties of a scale of competencies in online teaching, which represents a very valuable contribution to the country’s Higher Education System, especially to the distance modality in its online expression which, due to its incipient development in the country, is positioned as one of the modalities of higher education that has been little explored and/or evaluated. Its application in the HEIs participating in its validation is completely valid and the results derived from research or evaluation processes will provide reliable and objective information on the competencies of teachers who facilitate teaching processes in online environments, although these should be taken with great caution.
This scale represents for the Higher Education System of the country a model for the evaluation of teaching competencies in online environments with a high potential to provide sufficient elements for the issuance of technical and professional judgments in a valid and reliable manner. Therefore, this instrument establishes a first base and constitutes a first approach to the evaluation of competencies in online teaching in university professors in the Honduran context. Thus, the instrument will contribute to generate knowledge and fill multiple gaps in knowledge about its measurable construct and, in aggregate, about this modality that represents a valuable and innovative alternative to achieve a greater coverage of higher education in our country.
The evaluation model, therefore, is positioned as a guideline for the monitoring and follow-up of the quality assurance of higher education in distance modality in its online expression in the diverse HEIs that apply this modality in the country, focusing on the knowledge, skills and abilities that teachers who develop instructional practices under this modality should possess. The evaluation model can also serve as a referential framework for the evaluation of teaching performance in the pedagogical spaces that are developed in online environments so that such evaluation is contextualized to the distance modality in its online expression. This would imply a significant advance in teacher evaluation in the online modality by putting an end to the practice of evaluating teachers who teach in this modality in the same way as those who teach in face-to-face environments when their approaches and demands vary from one another.
On the other hand, this scale represents a contextualized input for the development of research related to the construct measured by the scale or, as it was originally conceived, as an evaluation instrument to assess competencies in online teaching. Because of its psychometric quality, its applicability in either of these two contexts will allow researchers, evaluators or competent authorities to efficiently identify teachers’ strengths and opportunities for improvement in terms of their teaching competencies. Therefore, the instrument will provide objective information/results that will lead, at the institutional level, to the generation of training and professionalization spaces for teachers to develop or enhance their competencies and, at the higher education system level, to the generation of significant inputs for the creation of educational policies in order to optimize teaching practices in the online modality.
Finally, for university professors who teach in online environments, this instrument represents a useful input to promote a process of individualized self-reflection that allows them to identify their own strengths in order to enhance them and also their areas for improvement so that their teaching work is framed in the search for and assurance of quality in the distance modality in its online expression. On future occasions when there is an abrupt transition from the traditional face-to-face modality to the online modality, or for novice teachers who are entering this modality for the first time, the instrument will serve as a methodological guide on the knowledge, skills and abilities that they should possess in order to provide quality, meaningful and efficient teaching in online environments.
REFERENCES
Abad, F. J., Olea, J., Ponsoda, V., & García, C. (2011). Medición en ciencias sociales y de la salud. Editorial Síntesis.
Albrahim, F. (2020). Online Teaching Skills and Competencies. TOJET: The Turkish Online Journal of Educational Technology, 19(1), 9-20.
Cadena, L. A., & Ramos, L. L. (2023). Pandemia y educación superior en América Latina. Revista de la Educación Superior, 52(205), 1-20. https://doi.org/10.36857/resu.2023.205.2367
Carbonell, C. E., Rodríguez, R., Sosa, L. A., & Alva, M. A. (2021). De la educación a distancia en pandemia a la modalidad hibrida en pospandemia. Revista Venezolana de Gerencia, 26(96), 1154-1171. https://doi.org/10.52080/rvgluz.26.96.10
Dirección de Educación Superior (DES). (14 de agosto de 2014). Reglamento de la Educación a Distancia en el Nivel de Educación Superior de Honduras. Acuerdo n° 2572-264-2012. DO: La Gaceta.
Elosua, P., Aguado, D., Fonseca-Pedrero, E., Abad, F. J., & Santamaría, P. (2023). New Trends in Digital Technology-Based Psychological and Educational Assessment. Psicothema, 35(1), 50-57. https://doi.org/10.7334/psicothema2022.241
Farmer, H., & Ramsdale, J. (2016). Teaching Competencies for the Online Environment. Canadian Journal of Learning and Technology, 42(3), 1-17. https://doi.org/10.21432/T2V32J
Ferrando, P. J., Lorenzo-Seva, U., Hernández-Dorado, A., & Muñiz, J. (2022). Decálogo para el Análisis Factorial de los Ítems de un Test. Psicothema, 34(1), 7-17. https://doi.org/10.7334/psicothema2021.456
Flora, D. B. (2020). Your Coefficient Alpha Is Probably Wrong, but Which Coefficient Omega Is Right? A Tutorial on Using R to Obtain Better Reliability Estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484-501. https://doi.org/10.1177/2515245920951747
Garcia-Pardina, A., Abad, F. J., Christensen, A. P., Golino, H., & Garrido, L. E. (2024). Dimensionality assessment in the presence of wording effects: A network psychometric and factorial approach. Behavior Research Methods. https://doi.org/10.3758/s13428-024-02348-w
Grieder, S., & Steiner, M. D. (2021). Algorithmic jingle jungle: A comparison of implementations of principal axis factoring and promax rotation in R and SPSS. Behavior Research Methods, 54, 54-74. https://doi.org/10.3758/s13428-021-01581-x
Instituto Nacional de Estadística. (2019). LXV Encuesta Permanente de Hogares de Propósitos Múltiples. Junio del 2019.
Instituto Nacional de Estadística. (2023). LXXVII Encuesta Permanente de Hogares de Propósitos Múltiples. Junio de 2023.
International Commission on the Futures of Education. (2020). Education in a post-COVID world: Nine ideas for public action. UNESCO.
Izquierdo, I., Olea, J., & Abad, F. (2014). Exploratory factor analysis in validation studies: Uses and recommendations. Psicothema, 26(3), 395-400. https://doi.org/10.7334/psicothema2013.349
Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El Análisis Factorial Exploratorio de los Ítems: una guía práctica, revisada y actualizada. Anales de Psicología, 30(3), 1151-1169. https://doi.org/10.6018/analesps.30.3.199361
Meinck, S., Fraillon, J., & Strietholt, R. (Eds.) (2022). The impact of the COVID-19 pandemic on education. UNESCO.
Mullo, H., & Marcatoma, J. (2022). Introducción a la Psicometría y Estadística. Editorial Grupo Compás.
Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test. Psicothema, 31(1), 7-16. https://doi.org/10.7334/psicothema2018.291
Organization for Economic Cooperation and Development. (2021). The State of Higher Education: One Year into the COVID-19 Pandemic. OECD Publishing. https://doi.org/10.1787/83c41957-en
Quintanilla, M. (2016). La educación superior a distancia y virtual en Honduras: avances y desafíos. In M. Cruz & C. Rama (Eds.), La educación a distancia y virtual en Centroamérica y El Caribe (pp. 71-91). Ediciones UAPA.
Rico-Reintsch, K. I. (2019). Uso de autoevaluación docente como herramienta innovadora para el mejoramiento de las asignaturas universitarias. Revista CEA, 5(10), 69-81. https://doi.org/10.22430/24223182.1445
Rigo, D. Y., & Donolo, D. (2019). Modelos de ecuaciones estructurales usos en investigación psicológica y educativa. Revista Interamericana de Psicología, 52(3), 345-357. https://doi.org/10.30849/ripijp.v52i3.388
UNESCO. (2017). La Educación superior virtual en América Latina y El Caribe. UNESCO IESALC.
Universidad Nacional Autónoma de Honduras. (2009). Modelo Educativo de la Universidad Nacional Autónoma de Honduras. Serie de Publicaciones de la Reforma Universitaria No. 3.
Universidad Nacional Autónoma de Honduras. (2014). La Educación a Distancia en la Universidad Nacional Autónoma de Honduras. Serie de Publicaciones de la Reforma Universitaria No. 5.
Universidad Nacional Autónoma de Honduras. (2021). Anuario Estadístico N° 51.
Universidad Pedagógica Nacional Francisco Morazán. (2022). Memoria Estadística UPNFM 2022.
Ventura-León, J., & Caycho-Rodríguez, T. (2017). El coeficiente Omega: un método alternativo para la estimación de la confiabilidad. Revista Latinoamericana de Ciencias Sociales, Niñez y Juventud, 15(1), 625-627.
Yusoff, M. (2019). ABC of content validation and content validity index calculation. Education in Medicine Journal, 11(2), 49-54. https://doi.org/10.21315/eimj2019.11.2.6