ChatGPT assistants in online higher education and student satisfaction: a case study

Authors

DOI:

https://doi.org/10.5944/ried.28.2.43552

Keywords:

artificial intelligence, chatbot, ChatGPT, higher education, online teaching, technology

Abstract

The perception and satisfaction of students about a virtual assistant based on OpenAI ChatGPT 3.5, integrated in 21 different subjects of the virtual campus of an online university, have been analyzed in this study. Using a mixed methodological approach, information was collected on a sample of 391 students using the validated COMUNICA questionnaire, which included four constructs: Virtual Assistant Efficiency, Learning Impact, Skill Development, and Technical and Accessibility Aspects. The analysis included descriptive statistics, inferential statistical tests, Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), complemented by a qualitative analysis of student and teacher comments. The quantitative results highlight that the female gender values the effectiveness of the assistant more than the male gender. The CFA confirmed that the factors can be grouped under a single latent variable: student satisfaction. In addition, the efficiency of the virtual assistant was found to be the most significant factor in the perception of student satisfaction, followed by the impact on learning, skill development and technical aspects. The qualitative analysis revealed mostly positive perceptions, highlighting the usefulness of the assistant in learning, an interest in extending its use to other subjects and suggestions for improvement in the accuracy of answers and functionality. It is concluded that virtual assistants have a positive impact on higher education, optimizing autonomous learning and educational interaction, although technical and design challenges persist that limit their full potential.

Downloads

Author Biography

Miguel-Ángel Cabeza-Rodríguez, Universidad Francisco de Vitoria, UFV (Spain)

Ph.D. in Physics, a Master’s in Business Administration, and is an Electronic Engineer. He is currently a faculty member in the Physics undergraduate program at the International University of La Rioja, where he also serves as Head of Laboratories for the Proeduca Group. His main research interests include Learning and Knowledge Technologies (LKT), and the theory of adult learning (andragogy).

References

Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152

Ait Baha, T., El Hajji, M., Es-Saady, Y., & Fadili, H. (2024). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29(8), 10153-10176. https://doi.org/10.1007/s10639-023-12166-w

Al-Emran, M., AlQudah, A. A., Abbasi, G. A., Al-Sharafi, M. A., & Iranmanesh, M. (2024). Determinants of Using AI-Based Chatbots for Knowledge Sharing: Evidence From PLS-SEM and Fuzzy Sets (fsQCA). IEEE Transactions on Engineering Management, 71, 4985-4999. https://doi.org/10.1109/TEM.2023.3237789

Anderson, T. W., & Darling, D. A. (1952). Asymptotic theory of certain "goodness of fit" criteria based on stochastic processes. The Annals of Mathematical Statistics, 23(2), 193-212. https://doi.org/10.1214/aoms/1177729437

Bagozzi, R. P., & Yi, Y. (1989). The Degree of Intention Formation as a Moderator of the Attitude-Behavior Relationship. Social Psychology Quarterly, 52(4), 266. https://doi.org/10.2307/2786991

Bahrini, A., Khamoshifar, M., Abbasimehr, H., Riggs, R. J., Esmaeili, M., Majdabadkohne, R. M., & Pasehvar, M. (2023). ChatGPT: Applications, Opportunities, and Threats. 2023 Systems and Information Engineering Design Symposium (SIEDS), 274-279. https://doi.org/10.1109/SIEDS58326.2023.10137850

Bartlett, M. S. (1954). A Note on the Multiplying Factors for Various χ2 Approximations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 16(2), 296-298. https://doi.org/10.1111/j.2517-6161.1954.tb00174.x

Bonett, D. G. (2002). Sample Size Requirements for Testing and Estimating Coefficient Alpha. Journal of Educational and Behavioral Statistics, 27(4), 335-340. https://doi.org/10.3102/10769986027004335

Brown, T. A. (2015). Confirmatory factor analysis for applied research. The Guilford Press.

Browne, M. W., & Cudeck, R. (1992). Alternative Ways of Assessing Model Fit. Sociological Methods & Research, 21(2), 230-258. https://doi.org/10.1177/0049124192021002005

Cai, Z., Fan, X., & Du, J. (2017). Gender and attitudes toward technology use: A meta-analysis. Computers & Education, 105, 1-13. https://doi.org/10.1016/j.compedu.2016.11.003

Chang, D. H., Lin, M. P.-C., Hajian, S., & Wang, Q. Q. (2023). Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization. Sustainability, 15(17), 12921. https://doi.org/10.3390/su151712921

Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510

Chen, X., Xie, H., Zou, D., & Hwang, G.-J. (2020). Application and theory gaps during the rise of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence, 1, 100002. https://doi.org/10.1016/j.caeai.2020.100002

Cheng, Z., Zhang, Z., Xu, Q., Maeda, Y., & Gu, P. (2023). A meta-analysis addressing the relationship between self-regulated learning strategies and academic performance in online higher education. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-023-09390-1

Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences (0 ed.). Routledge. https://doi.org/10.4324/9780203771587

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Curran, P. J., Bollen, K. A., Chen, F., Paxton, P., & Kirby, J. B. (2003). Finite Sampling Properties of the Point Estimates and Confidence Intervals of the RMSEA. Sociological Methods & Research, 32(2), 208-252. https://doi.org/10.1177/0049124103256130

Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1206936. https://doi.org/10.3389/feduc.2023.1206936

Dwivedi, Y. K., Hughes, D. L., Coombs, C., Constantiou, I., Duan, Y., Edwards, J. S., Gupta, B., Lal, B., Misra, S., Prashant, P., Raman, R., Rana, N. P., Sharma, S. K., & Upadhyay, N. (2021). Impact of COVID-19 pandemic on information management research and practice: Transforming education, work and life. International Journal of Information Management, 55, 102211. https://doi.org/10.1016/j.ijinfomgt.2020.102211

Feuerriegel, S., & Pröllochs, N. (2023). Sentiment Analysis R Package (Versión 1.3-5) [R]. https://cran.r-project.org/web/packages/SentimentAnalysis/vignettes/SentimentAnalysis.htmlhttps://doi.org/10.1016/j.ijinfomgt.2020.102211

Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the new world of HCI. Interactions, 24(4), 38-42. https://doi.org/10.1145/3085558

Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968). The Discovery of Grounded Theory; Strategies for Qualitative Research. Nursing Research, 17(4), 364. https://doi.org/10.1097/00006199-196807000-00014

Grassini, S. (2023). Shaping the Future of Education: Exploring the Potential and Consequences of AI and ChatGPT in Educational Settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis (Eighth edition). Cengage.

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5

Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., Han, W., Huang, M., Jin, Q., Lan, Y., Liu, Y., Liu, Z., Lu, Z., Qiu, X., Song, R., … Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, 2, 225-250. https://doi.org/10.1016/j.aiopen.2021.08.002

Hattie, J. (2023). Visible Learning: The Sequel: A Synthesis of Over 2,100 Meta-Analyses Relating to Achievement (1.a ed.). Routledge. https://doi.org/10.4324/9781003380542

Haynes, S. N., Richard, D., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological assessment, 7(3), 238.

https://doi.org/10.1037/1040-3590.7.3.238

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

Hwang, G.-J., & Chang, C.-Y. (2023). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7), 4099-4112. https://doi.org/10.1080/10494820.2021.1952615

Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36. https://doi.org/10.1007/BF02291575

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274

Kline, R. B. (2023). Principles and Practice of Structural Equation Modeling (Fifth edition). Guilford.

Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973-1018. https://doi.org/10.1007/s10639-022-11177-3

Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1

Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024). GPT understands, too. AI Open, 5, 208-215. https://doi.org/10.1016/j.aiopen.2023.08.012

Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810-819. https://doi.org/10.1037/0022-3514.70.4.810

Motlagh, N. Y., Khajavi, M., Sharifi, A., & Ahmadi, M. (2023). The Impact of Artificial Intelligence on the Evolution of Digital Education: A Comparative Study of OpenAI Text Generation Tools including ChatGPT, Bing Chat, Bard, and Ernie (Versión 1). arXiv. https://doi.org/10.48550/ARXIV.2309.02029

Nandwani, P., & Verma, R. (2021). A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1), 81. https://doi.org/10.1007/s13278-021-00776-6

Nunnally, J. C. (1978). Psychometric theory (2d ed). McGraw-Hill.

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033

Onal, S., & Kulavuz-Onal, D. (2024). A Cross-Disciplinary Examination of the Instructional Uses of ChatGPT in Higher Education. Journal of Educational Technology Systems, 52(3), 301-324. https://doi.org/10.1177/00472395231196532

OpenAI. (s. f.). OpenAI API Reference. Recuperado 25 de noviembre de 2024, de https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature

Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549-1565. https://doi.org/10.1002/cae.22326

Pérez-Marín, D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms. Digital, 1(1), 18-33. https://doi.org/10.3390/digital1010002

Perrotta, C., & Selwyn, N. (2020). Deep learning goes to school: Toward a relational understanding of AI in education. Learning, Media and Technology, 45(3), 251-269. https://doi.org/10.1080/17439884.2020.1686017

Peters, M. A., Jackson, L., Papastephanou, M., Jandrić, P., Lazaroiu, G., Evers, C. W., Cope, B., Kalantzis, M., Araya, D., Tesar, M., Mika, C., Chen, L., Wang, C., Sturm, S., Rider, S., & Fuller, S. (2024). AI and the future of humanity: ChatGPT-4, philosophy and education – Critical responses. Educational Philosophy and Theory, 56(9), 828-862. https://doi.org/10.1080/00131857.2023.2213437

Pokhrel, S., & Chhetri, R. (2021). A Literature Review on Impact of COVID-19 Pandemic on Teaching and Learning. Higher Education for the Future, 8(1), 133-141. https://doi.org/10.1177/2347631120983481

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in nursing & health, 30(4), 459-467.

https://doi.org/10.1002/nur.20199

Polverini, G., & Gregorcic, B. (2024). How understanding large language models can inform the use of ChatGPT in physics education. European Journal of Physics, 45(2), 025701. https://doi.org/10.1088/1361-6404/ad1420

Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02

Roszkowski, M. J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35(1), 113-130. https://doi.org/10.1080/02602930802618344

Ruiz-Corbella, M., & García-Aretio, L. (2023). Virtual mobility in higher education, ¿chance or utopy? Revista Española de Pedagogía, 68(246). https://doi.org/10.22550/2174-0909.3568

Streiner, D. L. (2003). Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency. Journal of Personality Assessment, 80(1), 99-103. https://doi.org/10.1207/S15327752JPA8001_18

Taani, O., & Alabidi, S. (2024). ChatGPT in education: Benefits and challenges of ChatGPT for mathematics and science teaching practices. International Journal of Mathematical Education in Science and Technology, 1-30. https://doi.org/10.1080/0020739X.2024.2357341

Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models (Versión 1). arXiv. https://doi.org/10.48550/ARXIV.2102.02503

Thottoli, M. M., Alruqaishi, B. H., & Soosaimanickam, A. (2024). Robo academic advisor: Can chatbots and artificial intelligence replace human interaction? Contemporary Educational Technology, 16(1), ep485. https://doi.org/10.30935/cedtech/13948

Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x

Venkatesh, V., & Morris, M. G. (2000). Why Don’t Men Ever Stop to Ask for Directions? Gender, Social Influence, and Their Role in Technology Acceptance and Usage Behavior. MIS Quarterly, 24(1), 115. https://doi.org/10.2307/3250981

Vierhauser, M., Groher, I., Antensteiner, T., & Sauerwein, C. (2024). Towards Integrating Emerging AI Applications in SE Education. 2024 36th International Conference on Software Engineering Education and Training (CSEE&T), 1-5. https://doi.org/10.1109/CSEET62301.2024.10663045

Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716

Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2286. https://doi.org/10.29333/ejmste/13272

Wheaton, B. (1987). Assessment of Fit in Overidentified Models with Latent Variables. Sociological Methods & Research, 16(1), 118-154. https://doi.org/10.1177/0049124187016001005

White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. http://arxiv.org/abs/2302.11382

Winkler, R., & Soellner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Academy of Management Proceedings, 2018(1), 15903. https://doi.org/10.5465/AMBPP.2018.15903abstract

Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample Size Requirements for Structural Equation Models: An Evaluation of Power, Bias, and Solution Propriety. Educational and Psychological Measurement, 73(6), 913-934. https://doi.org/10.1177/0013164413495237

Yung, Y., & Bentler, P. M. (1994). Bootstrap‐corrected ADF test statistics in covariance structure analysis. British Journal of Mathematical and Statistical Psychology, 47(1), 63-84. https://doi.org/10.1111/j.2044-8317.1994.tb01025.x

Yurdugül, H. (2008). Minimum sample size for Cronbach's coefficient alpha: A Monte-Carlo study. Hacettepe Üniversitesi eğitim fakültesi dergisi, 35(35), 1-9.

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., … Wen, J.-R. (2023). A Survey of Large Language Models (Versión 15). arXiv. https://doi.org/10.48550/ARXIV.2303.18223

Published

2025-04-30

How to Cite

Cabeza-Rodríguez, M.- Ángel. (2025). ChatGPT assistants in online higher education and student satisfaction: a case study. RIED-Revista Iberoamericana de Educación a Distancia, 28(2), 9–38. https://doi.org/10.5944/ried.28.2.43552

Similar Articles

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 > >> 

You may also start an advanced similarity search for this article.