AI-ED-SAT: design and validation of a questionnaire for self-assessment of teaching skills in educational AI
DOI:
https://doi.org/10.5944/ried.45413Keywords:
artificial intelligence, education, teacher competencies, questionnaire, instrument validationAbstract
The accelerated integration of Artificial Intelligence (AI) in education poses new challenges for teacher training and assessment. This study presents the design, validation, and psychometric analysis of the AI-ED-SAT questionnaire, a self-assessment tool designed for teachers to diagnose their level of preparedness in the pedagogical, ethical, and curricular use of AI. The AI‑ED‑SAT questionnaire was validated within the Design‑Based Research approach. It comprises four dimensions aligned with UNESCO’s current AI competency frameworks (2024a, 2024b): conceptual understanding of AI, pedagogical use of intelligent tools, ethical and critical reflection, and curricular integration. Its construction was based on an exhaustive theoretical review and an iterative validation process using the Delphi method with 12 experts in educational technology, AI, and teacher training. Subsequently, in a pilot test involving a sample of 128 teachers from various educational levels, the reliability (Cronbach’s α = 0.93), content validity, and construct validity were analyzed. The results of the exploratory factor analysis (EFA) confirmed the grouping of items into four theoretical factors, while the confirmatory factor analysis (CFA) showed excellent fit indices (CFI = 0.96; RMSEA = 0.045). The AI‑ED‑SAT is a robust and up‑to‑date tool that is useful in both research and teacher training programs. Its self-reflective approach helps strengthen teachers’ critical literacy and professional agency in the face of the challenges posed by AI in education.
Downloads
References
Alier Forment, M., García Peñalvo, F. J., Casañ Guerrero, M. J., Pereira, J. A., & Llorens-Largo, F. (2024, 8 October). Safe AI in Education Manifesto (Version 0.4.0). Safe AI in Education Manifesto. https://manifesto.safeaieducation.org
Ayuso del Puerto, D., & Gutiérrez Esteban, P. (2022). La inteligencia artificial como recurso educativo durante la formación inicial del profesorado. RIED-Revista Iberoamericana de Educación a Distancia, 25(2), 347-362. https://doi.org/10.5944/ried.25.2.32332
Baltazar, C. (2023). Herramientas de IA aplicables a la educación. Technology Rain Journal, 2(2), e15-e15. https://doi.org/10.55204/trj.v2i2.e15
Bo, N. S. W. (2024). OECD digital education outlook 2023: Towards an effective education ecosystem. Hungarian Educational Research Journal, 15(2), 284-289. https://doi.org/10.1556/063.2024.00340
Borge, R., García, J., Oliver, R., & Salomón, L. (2005). Competencias y diseño de la evaluación continua y final en el Espacio Europeo de Educación Superior. Dirección General de Universidades, MEC.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178. https://doi.org/10.1207/s15327809jls0202_2
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.
Cabero-Almenara, J., Barroso-Osuna, J., Palacios-Rodríguez, A., & Llorente-Cejudo, C. (2020). Marcos de competencias digitales para docentes universitarios: Su evaluación a través del coeficiente competencia experta. Revista Electrónica Interuniversitaria de Formación del Profesorado, 23(3), 17–34. https://doi.org/10.6018/reifop.414501
Caena, F., & Redecker, C. (2019). Aligning teacher competence frameworks to 21st century challenges: The case for the European Digital Competence Framework for Educators (DigCompEdu). European Journal of Education, 54(3), 356-369. https://doi.org/10.1111/ejed.12345
Celik, I. (2023). Towards Intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethically integrate artificial intelligence (AI)-based tools into education. Computers in Human Behavior, 138, 107468. https://doi.org/10.1016/j.chb.2022.107468
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
Chesterman, S. (2021). Through a glass, darkly: Artificial intelligence and the problem of opacity. The American Journal of Comparative Law, 69(2), 271-294. https://doi.org/10.1093/ajcl/avab012
Chiu, T. K., Ahmad, Z., Ismailov, M., & Sanusi, I. T. (2024). What are artificial intelligence literacy and competency? A comprehensive framework to support them. Computers and Education Open, 6, 100171. https://doi.org/10.1016/j.caeo.2024.100171
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(1), 1-9. https://doi.org/10.7275/jyj1-4868
Delgado, N., Carrasco, L. C., de la Maza, M. S., & Etxabe-Urbieta, J. M. (2024). Aplicación de la inteligencia artificial (IA) en educación: Los beneficios y limitaciones de la IA percibidos por el profesorado de educación primaria, educación secundaria y educación superior. Revista Electrónica Interuniversitaria de Formación del Profesorado, 27(1), 207-224. https://doi.org/10.6018/reifop.577211
Diefes-Dux, H. A., Zawojewski, J. S., & Hjalmarson, M. A. (2010). Using educational research in the design of evaluation tools for open-ended problems. International Journal of Engineering Education, 26(4), 807-819.
Ferrante, E. (2021). Inteligencia artificial y sesgos algorítmicos: ¿Por qué deberían importarnos? Nueva Sociedad, (294), 27-36. https://nuso.org/articulo/inteligencia-artificial-y-sesgos-algoritmicos/
Flores Vivar, J. M., & García Peñalvo, F. J. (2023). Reflexiones sobre la ética, potencialidades y retos de la inteligencia artificial en el marco de la Educación de Calidad (ODS4). Comunicar: Revista Científica de Comunicación y Educación, (74), 37-47. https://doi.org/10.3916/C74-2023-03
Gallant, D. J., & Luthy, N. (2020). Mixed methods research in designing an instrument for consumer-oriented evaluation. Journal of MultiDisciplinary Evaluation, 16(34), 21-43. https://doi.org/10.56645/jmde.v16i34.583
García Peñalvo, F. J., Llorens-Largo, F., & Vidal, J. (2024). La nueva realidad de la educación ante los avances de la inteligencia artificial generativa. RIED-Revista Iberoamericana de Educación a Distancia, 27(1), 9-39. https://doi.org/10.5944/ried.27.1.37716
Ghomi, M., & Redecker, C. (2019, May). Digital competence of educators (DigCompEdu): Development and evaluation of a self-assessment instrument for teachers’ digital competence. In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019) (Vol. 1, pp. 541-548). SCITEPRESS. https://doi.org/10.5220/0007679005410548
Grand-Guillaume-Perrenoud, J. A., Geese, F., Uhlmann, K., Blasimann, A., Wagner, F. L., Neubauer, F. B., Huwendiek, S., Hahn, S., & Schmitt, K.-U. (2023). Mixed methods instrument validation: Evaluation procedures for practitioners developed from the validation of the Swiss Instrument for Evaluating Interprofessional Collaboration. BMC Health Services Research, 23, 83. https://doi.org/10.1186/s12913-023-09040-3
Hernández-Sampieri, R., Fernández-Collado, C., & Baptista-Lucio, P. (2006). Metodología de la investigación (4th ed.). McGraw-Hill Interamericana.
Hogarty, K. Y., Hines, C. V., Kromrey, J. D., Ferron, J. M., & Mumford, K. R. (2005). The quality of factor solutions in exploratory factor analysis: The influence of sample size, communality, and overdetermination. Educational and Psychological Measurement, 65(2), 202-226. https://doi.org/10.1177/0013164404267287
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Journal of Education, 57(4), 542-570. https://doi.org/10.1111/ejed.12533
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
Huang, L. (2023). Ethics of artificial intelligence in education: Student privacy and data protection. Science Insights Education Frontiers, 16(2), 2577-2587. https://doi.org/10.15354/sief.23.re202
Khreisat, M. N., Khilani, D., Rusho, M. A., Karkkulainen, E. A., Tabuena, A. C., & Uberas, A. D. (2024). Ethical implications of AI integration in educational decision making: Systematic review. Educational Administration: Theory and Practice, 30(5), 8521-8527. https://doi.org/10.53555/kuey.v30i5.4406
Lane, M., Williams, M., & Broecke, S. (2023). The impact of AI on the workplace: Main findings from the OECD AI surveys of employers and workers (OECD Social, Employment and Migration Working Papers, No. 288). OECD Publishing. https://doi.org/10.1787/ea0a0fe1-en
Linstone, H. A., & Turoff, M. (Eds.). (2002). The Delphi method: Techniques and applications. Addison-Wesley.
Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorio de los ítems: Una guía práctica, revisada y actualizada. Anales de Psicología, 30(3), 1151-1169. https://doi.org/10.6018/analesps.30.3.199361
Luckin, R. (2017). Towards artificial intelligence-based assessment systems. Nature Human Behaviour, 1(3), 0028. https://doi.org/10.1038/s41562-016-0028
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99. https://doi.org/10.1037/1082-989X.4.1.84
Mauri, T., Coll, C., & Onrubia, J. (2007). La evaluación de la calidad de los procesos de innovación docente universitaria: Una perspectiva constructivista. Revista de Docencia Universitaria, 5(1). https://doi.org/10.4995/redu.2007.6290
McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410601087
Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test. Psicothema, 31(1), 7–16. https://doi.org/10.7334/psicothema2018.291
Ng, D. T. K., Leung, J. K. L., Su, J., Ng, R. C. W., & Chu, S. K. W. (2023). Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educational Technology Research and Development, 71(1), 137-161. https://doi.org/10.1007/s11423-023-10203-6
OECD. (2022). Trends shaping education 2022. OECD Publishing. https://doi.org/10.1787/6ae8771a-en
Owan, V. J., Abang, K. B., Idika, D. O., Etta, E. O., & Bassey, B. A. (2023). Exploring the potential of artificial intelligence tools in educational measurement and assessment. Eurasia Journal of Mathematics, Science and Technology Education, 19(8), em2307. https://doi.org/10.29333/ejmste/13428
Panadero, E., Broadbent, J., Boud, D., & Lodge, J. M. (2019). Using formative assessment to influence self- and co-regulated learning: The role of evaluative judgement. European Journal of Psychology of Education, 34, 535-557. https://doi.org/10.1007/s10212-018-0407-8
Paulin, A. M., Barriga-Arceo, F. D., Mendiola, M. S., & González, A. M. (2024). Evidencias de validez de un instrumento para evaluar la competencia digital docente en educación médica. Investigación en Educación Médica, 13(51), 82-92. https://doi.org/10.22201/fm.20075057e.2024.51.23584
Rachbauer, T., Graup, J., & Rutter, E. (2025). Digital literacy and artificial intelligence literacy in teacher training. Forum for Education Studies, 3(1), 1842. https://doi.org/10.59400/fes1842
Ramírez, J. L. M. (2019). El proceso de elaboración y validación de un instrumento de medición documental. Acción y Reflexión Educativa, 44, 50-63. https://revistas.up.ac.pa/index.php/accion_reflexion_educativa/article/view/673
Ramírez-Montoya, M. S., & Lugo-Ocando, J. (2020). Revisión sistemática de métodos mixtos en el marco de la innovación educativa. Comunicar: Revista Científica de Comunicación y Educación, 28(65), 9–20. https://doi.org/10.3916/C65-2020-01
Reguant-Álvarez, M., & Torrado-Fonseca, M. (2016). El método Delphi. REIRE. Revista d’Innovació i Recerca en Educació, 9(1), 87-102. https://doi.org/10.1344/reire2016.9.1916
Sampieri, R. H. (2018). Metodología de la investigación: Las rutas cuantitativa, cualitativa y mixta. McGraw-Hill México.
Selwyn, N. (2019). Should robots replace teachers? AI and the future of education. John Wiley & Sons.
Sharma, D. M., Ramana, K. V., Jothilakshmi, R., Verma, R., Maheswari, B. U., & Boopathi, S. (2024). Integrating generative AI into K-12 curriculums and pedagogies in India: Opportunities and challenges. In M. S. P. Subathra & G. V. S. N. R. V. Prasad (Eds.), Facilitating global collaboration and knowledge sharing in higher education with generative AI (pp. 133-161). IGI Global. https://doi.org/10.4018/979-8-3693-0487-7.ch006
Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education: Research, 6(1), 1-21. https://doi.org/10.28945/199
UNESCO. (2021). Artificial intelligence and the futures of learning: Towards an ethical and inclusive approach. UNESCO. https://www.unesco.org/en/digital-education/ai-future-learning
UNESCO. (2023). Generative AI and the future of education. UNESCO. https://doi.org/10.54675/HOXG8740
UNESCO. (2024a). AI competency framework for students. UNESCO. https://doi.org/10.54675/JKJB9835
UNESCO. (2024b). AI competency framework for teachers. UNESCO. https://doi.org/10.54675/ZJTE2084
Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806-838. https://doi.org/10.1177/0011000006288127
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education. International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Andresa Sartor-Harada, Juliana Azevedo-Gomes
This work is licensed under a Creative Commons Attribution 4.0 International License.
The articles that are published in this journal are subject to the following terms:
1. The authors grant the exploitation rights of the work accepted for publication to RIED, guarantee to the journal the right to be the first publication of research understaken and permit the journal to distribute the work published under the license indicated in point 2.
2. The articles are published in the electronic edition of the journal under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. You can copy and redistribute the material in any medium or format, adapt, remix, transform, and build upon the material for any purpose, even commercially. You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
3. Conditions for self-archiving. Authors are encouraged to disseminate electronically the OnlineFirst version (assessed version and accepted for publication) of its articles before publication, always with reference to its publication by RIED, favoring its circulation and dissemination earlier and with this a possible increase in its citation and reach among the academic community.
