Marjan Mernik

The use of large language models with chatbots like ChatGPT has become increasingly popular among students, especially in Computer Science education. However, significant debates exist in the education community on the role of ChatGPT in learning. Therefore, it is critical to understand the potential impact of ChatGPT on the learning, engagement, and overall success of students in classrooms. In this empirical study, we report on a controlled experiment with 182 participants in a first-year undergraduate course on object-oriented programming. Our differential study divided students into two groups, one using ChatGPT and the other not using it for practical programming assignments. The study results showed that the students’ performance is not influenced by ChatGPT usage (no statistical significance between groups with a p-value of 0.730), nor are the grading results of practical assignments (p-value 0.760) and midterm exams (p-value 0.856). Our findings from the controlled experiment suggest that it is safe for novice programmers to use ChatGPT if specific measures and adjustments are adopted in the education process.

Our previous study showed that automatically generated attribute grammars were harder to comprehend than manually written attribute grammars, mostly due to unexpected solutions. This study is an internally differentiated replication of the previous experiment, but, unlike the previous one, it focused on testing the influence of code bloat on comprehension correctness and efficiency. While the experiment’s context, design, and measurements were kept mostly the same as in the original experiment, more realistic code bloat examples were introduced. The replicated experiment was conducted with undergraduate students from two universities, showing statistically significant differences in comprehension correctness and efficiency between attribute grammars without code bloat and attribute grammars with code bloat, although the participants perceived attribute grammars with code bloat as simple as attribute grammars without code bloat. On the other hand, there was no statistically significant difference in comprehension correctness and efficiency between automatically generated attribute grammars with possible unexpected solutions and attribute grammars with code bloat, although there was a statistically significant difference in participants’ perspective of simplicity between automatically generated attribute grammars with possible unexpected solutions and attribute grammars with code bloat. The participants perceived attribute grammars with code bloat as significantly simpler than automatically generated attribute grammars.