Higher Order Thinking and AI

This week I would like to share a recent SoTL article entitled, “How Higher-Order Thinking Shapes Student Attitudes Toward Machine Learning and AI Chatbots” by Pellas (2025). The author explores how cognitive skills, specifically creativity, critical thinking, and problem-solving mediate or moderate the relationship between academic achievement and attitudes toward machine learning. Drawing on data from 416 undergraduate students, the research offers one of the most empirically grounded examinations of how AI chatbots can enhance or reshape students’ cognitive engagement and perceptions of AI-supported education.

The study is situated within the broader discourse of AI-enhanced learning and cognitive development, emphasizing the convergence of machine learning and higher-order thinking in human–AI interaction. The work builds on prior research (e.g., Bansal et al., 2024; Huang et al., 2024) showing that AI chatbots not only facilitate learning but also influence metacognition and engagement. Theoretically, the study aligns with constructivist and cognitive theories that position learners as co-creators of knowledge, leveraging AI to extend but not replace human cognition.

Two hypotheses guided the inquiry:

1. Creativity mediates the relationship between academic achievement and attitudes toward AI/machine learning.

2. Critical thinking and problem-solving moderate this relationship, influencing how strongly academic performance predicts positive attitudes toward AI.

A cross-sectional design was used with students from multiple disciplines, all of whom had prior experience designing AI chatbot projects. Participants completed validated instruments measuring academic achievement, attitudes toward machine learning (Hopcan et al., 2024), and higher-order thinking (Hwang et al., 2018). Using path analysis and structural equation modeling (SEM), the study examined direct, mediating, and moderating relationships among these constructs.

Key Findings

• Creativity significantly mediated the relationship between academic achievement and attitudes toward machine learning (β = .33, p < .001). Students with higher creativity levels displayed more positive perceptions of AI.

• Critical Thinking and Problem-Solving as Moderators: These skills did not mediate the relationship but significantly moderated it (β = −.21 and −.13, p < .001), meaning they influenced how academic achievement translated into attitudes toward AI.

• Creativity served as a bridge between performance and positive engagement, while critical thinking and problem-solving functioned as filters that shape the depth and skepticism of engagement.

The results demonstrate that AI engagement is not cognitively neutral, while chatbots can stimulate learning, their educational impact depends on how students apply higher-order cognitive skills.The author concludes that creativity is the most influential cognitive driver of students’ openness toward AI, whereas critical thinking and problem-solving exert conditional influences. These findings extend prior research (Jia & Tu, 2024) by showing that creativity functions as a key motivational and cognitive mediator in AI-supported learning. Pedagogically, the study calls for instructional designs that balance creative exploration with critical evaluation. Faculty should integrate AI-based project-based learning (PBL) and reflective tasks that cultivate both creativity and cognitive discernment. The author also recommends blending AI chatbot interactions with human mentoring, ensuring that AI acts as a catalyst for higher-order thinking rather than a substitute for it.

References

Pellas, N. (2025). The role of students’ higher-order thinking skills in the relationship between academic achievements and machine learning using generative AI chatbots. Research and Practice in Technology Enhanced Learning, 20(36). https://doi.org/10.58459/rptel.2025.20036 

Bansal, G., Chamola, V., Hussain, A., Guizani, M., & Niyato, D. (2024). Transforming conversations with AI—A comprehensive study of ChatGPT. Cognitive Computation, 16(4), 2487–2510. https://doi.org/10.1007/s12559-023-10236-2 

Huang, F., Wang, Y., & Zhang, H. (2024). Modelling generative AI acceptance, perceived teachers’ enthusiasm and self‐efficacy to English as a foreign language learners’ well‐being in the digital era. European Journal of Education, 59(4), e12770. https://doi.org/10.1111/ejed.12770

Hwang, G. J., Lai, C. L., Liang, J. C., Chu, H. C., & Tsai, C. C. (2018). A long-term experiment to investigate the relationships between high school students’ perceptions of mobile learning and peer interaction and higher-order thinking tendencies. Educational Technology Research and Development, 66(1), 75–93. https://doi.org/10.1007/s11423-017-9540-3

Jia, X., & Tu, J. (2024). Towards a new conceptual model of AI-enhanced learning for college students: The roles of artificial intelligence capabilities, general self-efficacy, learning motivation, and critical thinking. Computers and Education: Artificial Intelligence, 6, 100230. https://doi.org/10.1016/j.caeai.2024.100230 

Lu, Y., Gao, J., & Zhang, X. (2024). Generative AI-assisted teaching skills training: Effects on preservice teachers’ self-efficacy and higher-order thinking. Teaching and Teacher Education, 139, 104550. https://doi.org/10.1016/j.tate.2024.104550