Unveiling the core constructs: A statistical approach to evaluating user experience with Chatbots in higher education (A case study from a university in Ecuador)

Jomar Elizabeth Guzman Seraquive, Patricio Álvarez-Muñoz, Kerly Palacios-Zamora, Dennis Alfredo Peralta Gamboa, Marco Faytong-Haro

Article ID: 6381
Vol 8, Issue 10, 2024

VIEWS - 117 (Abstract) 35 (PDF)

Abstract


Introduction: Chatbots are increasingly utilized in education, offering real-time, personalized communication. While research has explored technical aspects of chatbots, user experience remains under-investigated. This study examines a model for evaluating user experience and satisfaction with chatbots in higher education. Methodology: A four-factor model (information quality, system quality, chatbot experience, user satisfaction) was proposed based on prior research. An alternative two-factor model emerged through exploratory factor analysis, focusing on “Chatbot Response Quality” and “User Experience and Satisfaction with the Chatbot.” Surveys were distributed to students and faculty at a university in Ecuador to collect data. Confirmatory factor analysis validated both models. Results: The two-factor model explained a significantly greater proportion of the data’s variance (55.2%) compared to the four-factor model (46.4%). Conclusion: This study suggests that a simpler model focusing on chatbot response quality and user experience is more effective for evaluating chatbots in education. Future research can explore methods to optimize these factors and improve the learning experience for students.


Keywords


factor analysis; chatbot; higher education; virtual assistants; artificial intelligence; Turing test; user experience; satisfaction; response quality; algorithms; e-learning; simulation; multivalent statistics

Full Text:

PDF


References


Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006

Ait Baha, T., El Hajji, M., Es-Saady, Y., et al. (2023). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29(8), 10153–10176. https://doi.org/10.1007/s10639-023-12166-w

Baskara, F. R. (2023). Chatbots and flipped learning: Enhancing student engagement and learning outcomes through personalised support and collaboration. International Journal of Recent Educational Research, 4(2), 223–238.

Cao, C. C., Ding, Z., Lin, J., & Hopfgartner, F. (2023). AI Chatbots as Multi-Role Pedagogical Agents: Transforming Engagement in CS Education. arXiv, arXiv:2308.03992.

Chung, M., Ko, E., Joung, H., & Kim, S. J. (2020). Chatbot e-service and customer satisfaction regarding luxury brands. Journal of Business Research, 117, 587–595.

Deng, X., & Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4), 2940.

Fox, C. (2014). Chatbot takes on Turing. The Philosophers’ Magazine, 66, 8–8.

Liu, L., Subbareddy, R., & Raghavendra, C. (2022). Ai intelligence chatbot to improve students learning in the higher education platform. Journal of Interconnection Networks, 22(Supp02), 2143032.

López-Aguado, M., & Gutiérrez-Provecho, L. (2019). How to perform and interpret an exploratory factor analysis using SPSS Statistics. Revista d’Innovació i Recerca En Educació, 12(2), 1.

Malhotra, N. K., Nunan, D., & Birks, D. F. (2020). Marketing research. Pearson UK.

Martínez, C. M., & Sepúlveda, M. A. R. (2012). Introduction to exploratory factor analysis (Spanish). Revista Colombiana de Psiquiatría, 41(1), 197–207.

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033.

Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549–1565.

Peyton, K., & Unnikrishnan, S. (2023). A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs. Results in Engineering, 17, 100856.

Przegalinska, A., Ciechanowski, L., Stroz, A., et al. (2019). In bot we trust: A new methodology of chatbot performance measures. Business Horizons, 62(6), 785–797.

Sakulwichitsintu, S. (2023). ParichartBOT: a chatbot for automatic answering for postgraduate students of an open university. International Journal of Information Technology, 15(3), 1387–1397.

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862.

Trivedi, J. (2019). Examining the customer experience of using banking chatbots and its impact on brand love: The moderating role of perceived risk. Journal of Internet Commerce, 18(1), 91–111.

Zarouali, B., Van den Broeck, E., Walrave, M., & Poels, K. (2018). Predicting consumer responses to a chatbot on Facebook. Cyberpsychology, Behavior, and Social Networking, 21(8), 491–497.

Zhang, G., & Preacher, K. J. (2015). Factor rotation and standard errors in exploratory factor analysis. Journal of Educational and Behavioral Statistics, 40(6), 579–603.




DOI: https://doi.org/10.24294/jipd.v8i10.6381

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Jomar Elizabeth Guzman Seraquive, Patricio Álvarez-Muñoz, Kerly Palacios-Zamora, Dennis Alfredo Peralta Gamboa, Marco Faytong-Haro

License URL: https://creativecommons.org/licenses/by/4.0/

This site is licensed under a Creative Commons Attribution 4.0 International License.