Testing AI for Learning Impact

This week I would like to share ideas from a colleague at VU who has taken a SoTL approach to identify the impact of AI on his students. This is  not a scholarly, peer-reviewed article, it is from the Times Higher Ed but I am hoping you will appreciate the approach. The piece is entitled, “How to Test GenAI’s Impact on Learning” by Thibault Schrepel.

The author shares how he has integrated AI into his teaching and collecting data on how/if it is working as he intended it to in as turns AI into a learning lab, sharing six experiments:

1. Run Group Experiments. By dividing students into groups which include one barred from using AI;

   one allowed to use it without further instruction; and one trained in structured prompting and critique. The instructor found no differences in rote recall but significant gains in reasoning and writing among those taught structured prompting and critique. This approach supports cognitive information processing models, which distinguish between short-term recall and higher-order reasoning. While rote memory relies on rehearsal and retrieval practice (Anderson, 2010), reasoning benefits from scaffolding and metacognitive monitoring (Flavell, 1979).

2. Build an AI Research Assistant. Students develop AI-powered “daily digests” of scholarship, prompting models to synthesize course-related news. Comparing outputs exposes biases and prompt effects. This aligns with constructivist approaches (Bruner, 1966), where learners actively build knowledge. It also reflects dual coding theory (Paivio, 1986), as students process both textual content and AI-mediated summaries, reinforcing comprehension through multiple representational forms.

3. Compare Outputs. Students first summarize a dense text manually, then compare their work with outputs from multiple models. This highlights differences in detail, tone, and accuracy. Ties to cognitive load theory (Sweller, 1988)—AI can reduce extraneous load by speeding up comprehension, but students must evaluate accuracy to avoid cognitive shortcuts. The comparison process also strengthens metacognitive monitoring, a core mechanism in human learning.

4. Turn AI into a Socratic Partner. AI plays roles such as tutor, judge, or client, interrogating students’ arguments. This “study mode” transforms AI into a dialogic partner. This reflects Socratic pedagogy and social constructivism, emphasizing dialogue as a mode of knowledge construction. The back-and-forth questioning promotes deep processing (Craik & Lockhart, 1972), which enhances transfer and professional thinking.

5. Ask Them How It Feels. Structured reflection after AI use helps students articulate whether AI support felt empowering or displacing. Many welcomed mechanical help (grammar, citations) but resisted when AI generated core arguments. Reflection is essential to experiential learning theory (Kolb, 1984) and supports metacognition. Students begin to delineate the boundaries between human judgment and machine assistance, strengthening self-regulation (Pintrich, 2004).

6. Ban Uniform Bans. AI will neither rescue nor ruin higher ed. Instead, classrooms should be “laboratories” where faculty test, adapt, and share what works best. Academic freedom to experiment is essential. This resonates with pragmatism (Dewey, 1938), where education is viewed as inquiry in action. Pedagogical adaptability reflects the information processing model’s emphasis on feedback loops—faculty iterate based on what students actually learn, not abstract prescriptions.

References

Anderson, J. R. (2010). Cognitive psychology and its implications (7th ed.). Worth.

Bruner, J. S. (1966). Toward a theory of instruction. Harvard University Press.

Craik, F. I., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684.

Dewey, J. (1938). Experience and education. Macmillan.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911.

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice Hall.

Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.

Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self‐regulated learning in college students. Educational Psychology Review, 16(4), 385–407.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.