Assessing Reflective Learning through Human Revision of AI-Generated Essays

A Multi-Phase Study

Abstract

This study examined how university students utilize generative AI in the context of writing admissions essays and how the depth of their reflective thinking affects the quality of AI-assisted writing. One hundred twenty-six students participated in five types of writing tasks modeled on university application prompts, with varying levels of AI involvement. Each submission was blind-reviewed using a four-level rubric designed to capture finer distinctions in structure, logic, and expression. The results showed that, while the influence of initial writing ability was limited to the early stages of AI engagement, the depth of reflection—measured as the Reflection Depth Score (RDS)—was significantly associated with the quality of outputs across all tasks. Participants with high RDS demonstrated greater score improvement in later tasks, while those with low RDS sometimes experienced declines in performance. These findings suggest that the educational effectiveness of generative AI depends not only on its available skills but also on the learner's metacognitive abilities, underscoring the importance of reflective and dialogic processes in AI-integrated writing instruction.