ChatGPT assistants in online higher education and student satisfaction: a case study
DOI:
https://doi.org/10.5944/ried.28.2.43552Keywords:
artificial intelligence, chatbot, ChatGPT, higher education, online teaching, technologyAbstract
The perception and satisfaction of students about a virtual assistant based on OpenAI ChatGPT 3.5, integrated in 21 different subjects of the virtual campus of an online university, have been analyzed in this study. Using a mixed methodological approach, information was collected on a sample of 391 students using the validated COMUNICA questionnaire, which included four constructs: Virtual Assistant Efficiency, Learning Impact, Skill Development, and Technical and Accessibility Aspects. The analysis included descriptive statistics, inferential statistical tests, Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), complemented by a qualitative analysis of student and teacher comments. The quantitative results highlight that the female gender values the effectiveness of the assistant more than the male gender. The CFA confirmed that the factors can be grouped under a single latent variable: student satisfaction. In addition, the efficiency of the virtual assistant was found to be the most significant factor in the perception of student satisfaction, followed by the impact on learning, skill development and technical aspects. The qualitative analysis revealed mostly positive perceptions, highlighting the usefulness of the assistant in learning, an interest in extending its use to other subjects and suggestions for improvement in the accuracy of answers and functionality. It is concluded that virtual assistants have a positive impact on higher education, optimizing autonomous learning and educational interaction, although technical and design challenges persist that limit their full potential.
Downloads
References
Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
Ait Baha, T., El Hajji, M., Es-Saady, Y., & Fadili, H. (2024). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29(8), 10153-10176. https://doi.org/10.1007/s10639-023-12166-w
Al-Emran, M., AlQudah, A. A., Abbasi, G. A., Al-Sharafi, M. A., & Iranmanesh, M. (2024). Determinants of Using AI-Based Chatbots for Knowledge Sharing: Evidence From PLS-SEM and Fuzzy Sets (fsQCA). IEEE Transactions on Engineering Management, 71, 4985-4999. https://doi.org/10.1109/TEM.2023.3237789
Anderson, T. W., & Darling, D. A. (1952). Asymptotic theory of certain "goodness of fit" criteria based on stochastic processes. The Annals of Mathematical Statistics, 23(2), 193-212. https://doi.org/10.1214/aoms/1177729437
Bagozzi, R. P., & Yi, Y. (1989). The Degree of Intention Formation as a Moderator of the Attitude-Behavior Relationship. Social Psychology Quarterly, 52(4), 266. https://doi.org/10.2307/2786991
Bahrini, A., Khamoshifar, M., Abbasimehr, H., Riggs, R. J., Esmaeili, M., Majdabadkohne, R. M., & Pasehvar, M. (2023). ChatGPT: Applications, Opportunities, and Threats. 2023 Systems and Information Engineering Design Symposium (SIEDS), 274-279. https://doi.org/10.1109/SIEDS58326.2023.10137850
Bartlett, M. S. (1954). A Note on the Multiplying Factors for Various χ2 Approximations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 16(2), 296-298. https://doi.org/10.1111/j.2517-6161.1954.tb00174.x
Bonett, D. G. (2002). Sample Size Requirements for Testing and Estimating Coefficient Alpha. Journal of Educational and Behavioral Statistics, 27(4), 335-340. https://doi.org/10.3102/10769986027004335
Brown, T. A. (2015). Confirmatory factor analysis for applied research. The Guilford Press.
Browne, M. W., & Cudeck, R. (1992). Alternative Ways of Assessing Model Fit. Sociological Methods & Research, 21(2), 230-258. https://doi.org/10.1177/0049124192021002005
Cai, Z., Fan, X., & Du, J. (2017). Gender and attitudes toward technology use: A meta-analysis. Computers & Education, 105, 1-13. https://doi.org/10.1016/j.compedu.2016.11.003
Chang, D. H., Lin, M. P.-C., Hajian, S., & Wang, Q. Q. (2023). Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization. Sustainability, 15(17), 12921. https://doi.org/10.3390/su151712921
Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
Chen, X., Xie, H., Zou, D., & Hwang, G.-J. (2020). Application and theory gaps during the rise of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence, 1, 100002. https://doi.org/10.1016/j.caeai.2020.100002
Cheng, Z., Zhang, Z., Xu, Q., Maeda, Y., & Gu, P. (2023). A meta-analysis addressing the relationship between self-regulated learning strategies and academic performance in online higher education. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-023-09390-1
Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences (0 ed.). Routledge. https://doi.org/10.4324/9780203771587
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
Curran, P. J., Bollen, K. A., Chen, F., Paxton, P., & Kirby, J. B. (2003). Finite Sampling Properties of the Point Estimates and Confidence Intervals of the RMSEA. Sociological Methods & Research, 32(2), 208-252. https://doi.org/10.1177/0049124103256130
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1206936. https://doi.org/10.3389/feduc.2023.1206936
Dwivedi, Y. K., Hughes, D. L., Coombs, C., Constantiou, I., Duan, Y., Edwards, J. S., Gupta, B., Lal, B., Misra, S., Prashant, P., Raman, R., Rana, N. P., Sharma, S. K., & Upadhyay, N. (2021). Impact of COVID-19 pandemic on information management research and practice: Transforming education, work and life. International Journal of Information Management, 55, 102211. https://doi.org/10.1016/j.ijinfomgt.2020.102211
Feuerriegel, S., & Pröllochs, N. (2023). Sentiment Analysis R Package (Versión 1.3-5) [R]. https://cran.r-project.org/web/packages/SentimentAnalysis/vignettes/SentimentAnalysis.htmlhttps://doi.org/10.1016/j.ijinfomgt.2020.102211
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the new world of HCI. Interactions, 24(4), 38-42. https://doi.org/10.1145/3085558
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968). The Discovery of Grounded Theory; Strategies for Qualitative Research. Nursing Research, 17(4), 364. https://doi.org/10.1097/00006199-196807000-00014
Grassini, S. (2023). Shaping the Future of Education: Exploring the Potential and Consequences of AI and ChatGPT in Educational Settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis (Eighth edition). Cengage.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., Han, W., Huang, M., Jin, Q., Lan, Y., Liu, Y., Liu, Z., Lu, Z., Qiu, X., Song, R., … Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, 2, 225-250. https://doi.org/10.1016/j.aiopen.2021.08.002
Hattie, J. (2023). Visible Learning: The Sequel: A Synthesis of Over 2,100 Meta-Analyses Relating to Achievement (1.a ed.). Routledge. https://doi.org/10.4324/9781003380542
Haynes, S. N., Richard, D., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological assessment, 7(3), 238.
https://doi.org/10.1037/1040-3590.7.3.238
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
Hwang, G.-J., & Chang, C.-Y. (2023). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7), 4099-4112. https://doi.org/10.1080/10494820.2021.1952615
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36. https://doi.org/10.1007/BF02291575
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
Kline, R. B. (2023). Principles and Practice of Structural Equation Modeling (Fifth edition). Guilford.
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973-1018. https://doi.org/10.1007/s10639-022-11177-3
Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024). GPT understands, too. AI Open, 5, 208-215. https://doi.org/10.1016/j.aiopen.2023.08.012
Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810-819. https://doi.org/10.1037/0022-3514.70.4.810
Motlagh, N. Y., Khajavi, M., Sharifi, A., & Ahmadi, M. (2023). The Impact of Artificial Intelligence on the Evolution of Digital Education: A Comparative Study of OpenAI Text Generation Tools including ChatGPT, Bing Chat, Bard, and Ernie (Versión 1). arXiv. https://doi.org/10.48550/ARXIV.2309.02029
Nandwani, P., & Verma, R. (2021). A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1), 81. https://doi.org/10.1007/s13278-021-00776-6
Nunnally, J. C. (1978). Psychometric theory (2d ed). McGraw-Hill.
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
Onal, S., & Kulavuz-Onal, D. (2024). A Cross-Disciplinary Examination of the Instructional Uses of ChatGPT in Higher Education. Journal of Educational Technology Systems, 52(3), 301-324. https://doi.org/10.1177/00472395231196532
OpenAI. (s. f.). OpenAI API Reference. Recuperado 25 de noviembre de 2024, de https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549-1565. https://doi.org/10.1002/cae.22326
Pérez-Marín, D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms. Digital, 1(1), 18-33. https://doi.org/10.3390/digital1010002
Perrotta, C., & Selwyn, N. (2020). Deep learning goes to school: Toward a relational understanding of AI in education. Learning, Media and Technology, 45(3), 251-269. https://doi.org/10.1080/17439884.2020.1686017
Peters, M. A., Jackson, L., Papastephanou, M., Jandrić, P., Lazaroiu, G., Evers, C. W., Cope, B., Kalantzis, M., Araya, D., Tesar, M., Mika, C., Chen, L., Wang, C., Sturm, S., Rider, S., & Fuller, S. (2024). AI and the future of humanity: ChatGPT-4, philosophy and education – Critical responses. Educational Philosophy and Theory, 56(9), 828-862. https://doi.org/10.1080/00131857.2023.2213437
Pokhrel, S., & Chhetri, R. (2021). A Literature Review on Impact of COVID-19 Pandemic on Teaching and Learning. Higher Education for the Future, 8(1), 133-141. https://doi.org/10.1177/2347631120983481
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in nursing & health, 30(4), 459-467.
https://doi.org/10.1002/nur.20199
Polverini, G., & Gregorcic, B. (2024). How understanding large language models can inform the use of ChatGPT in physics education. European Journal of Physics, 45(2), 025701. https://doi.org/10.1088/1361-6404/ad1420
Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02
Roszkowski, M. J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35(1), 113-130. https://doi.org/10.1080/02602930802618344
Ruiz-Corbella, M., & García-Aretio, L. (2023). Virtual mobility in higher education, ¿chance or utopy? Revista Española de Pedagogía, 68(246). https://doi.org/10.22550/2174-0909.3568
Streiner, D. L. (2003). Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency. Journal of Personality Assessment, 80(1), 99-103. https://doi.org/10.1207/S15327752JPA8001_18
Taani, O., & Alabidi, S. (2024). ChatGPT in education: Benefits and challenges of ChatGPT for mathematics and science teaching practices. International Journal of Mathematical Education in Science and Technology, 1-30. https://doi.org/10.1080/0020739X.2024.2357341
Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models (Versión 1). arXiv. https://doi.org/10.48550/ARXIV.2102.02503
Thottoli, M. M., Alruqaishi, B. H., & Soosaimanickam, A. (2024). Robo academic advisor: Can chatbots and artificial intelligence replace human interaction? Contemporary Educational Technology, 16(1), ep485. https://doi.org/10.30935/cedtech/13948
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Venkatesh, V., & Morris, M. G. (2000). Why Don’t Men Ever Stop to Ask for Directions? Gender, Social Influence, and Their Role in Technology Acceptance and Usage Behavior. MIS Quarterly, 24(1), 115. https://doi.org/10.2307/3250981
Vierhauser, M., Groher, I., Antensteiner, T., & Sauerwein, C. (2024). Towards Integrating Emerging AI Applications in SE Education. 2024 36th International Conference on Software Engineering Education and Training (CSEE&T), 1-5. https://doi.org/10.1109/CSEET62301.2024.10663045
Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716
Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2286. https://doi.org/10.29333/ejmste/13272
Wheaton, B. (1987). Assessment of Fit in Overidentified Models with Latent Variables. Sociological Methods & Research, 16(1), 118-154. https://doi.org/10.1177/0049124187016001005
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. http://arxiv.org/abs/2302.11382
Winkler, R., & Soellner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Academy of Management Proceedings, 2018(1), 15903. https://doi.org/10.5465/AMBPP.2018.15903abstract
Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample Size Requirements for Structural Equation Models: An Evaluation of Power, Bias, and Solution Propriety. Educational and Psychological Measurement, 73(6), 913-934. https://doi.org/10.1177/0013164413495237
Yung, Y., & Bentler, P. M. (1994). Bootstrap‐corrected ADF test statistics in covariance structure analysis. British Journal of Mathematical and Statistical Psychology, 47(1), 63-84. https://doi.org/10.1111/j.2044-8317.1994.tb01025.x
Yurdugül, H. (2008). Minimum sample size for Cronbach's coefficient alpha: A Monte-Carlo study. Hacettepe Üniversitesi eğitim fakültesi dergisi, 35(35), 1-9.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., … Wen, J.-R. (2023). A Survey of Large Language Models (Versión 15). arXiv. https://doi.org/10.48550/ARXIV.2303.18223

Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Miguel-Ángel Cabeza-Rodríguez

This work is licensed under a Creative Commons Attribution 4.0 International License.
The articles that are published in this journal are subject to the following terms:
1. The authors grant the exploitation rights of the work accepted for publication to RIED, guarantee to the journal the right to be the first publication of research understaken and permit the journal to distribute the work published under the license indicated in point 2.
2. The articles are published in the electronic edition of the journal under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. You can copy and redistribute the material in any medium or format, adapt, remix, transform, and build upon the material for any purpose, even commercially. You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
3. Conditions for self-archiving. Authors are encouraged to disseminate electronically the OnlineFirst version (assessed version and accepted for publication) of its articles before publication, always with reference to its publication by RIED, favoring its circulation and dissemination earlier and with this a possible increase in its citation and reach among the academic community.