2024

Choice-based Personalization in MOOCs: Impact on Activity and Perceived Value

Ram I, Harris S, Roll I. Choice-based Personalization in MOOCs: Impact on Activity and Perceived Value. International Journal of Artificial Intelligence in Education. 2024 Jun;34(2):376-394. https://doi.org/10.1007/s40593-023-00334-5
 

Personalization in education describes instruction that is tailored to learners’ interests, attributes, or background and can be applied in various ways, one of which is through choice. In choice-based personalization, learners choose topics or resources that fit them the most. Personalization may be especially important (and under-used) with diverse learners, such as in a MOOC context. We report the impact of choice-based personalization on activity level, learning gains, and satisfaction in a Climate Science MOOC. The MOOC’s learning assignments had learners choose resources on climate-related issues in either their geographic locale (Personalized group) or in given regions (Generic group). 219 learners completed at least one of the two assignments. Over the entire course, personalization increased learners’ activity (number of course events), self-reported understanding of local issues, and self-reported likelihood to change climate-related habits. We found no differences on assignment completion rate, assignment length, and self-reported time-on-task. These results show that benefits of personalization extend beyond the original task and affect learners’ overall experience. We discuss design and implications of choice-based personalization, as well as opportunities for choice-based personalization at scale.

@article{aa1880fecfac43698410043d591f36ee,
title = "Choice-based Personalization in MOOCs: Impact on Activity and Perceived Value",
abstract = "Personalization in education describes instruction that is tailored to learners{\textquoteright} interests, attributes, or background and can be applied in various ways, one of which is through choice. In choice-based personalization, learners choose topics or resources that fit them the most. Personalization may be especially important (and under-used) with diverse learners, such as in a MOOC context. We report the impact of choice-based personalization on activity level, learning gains, and satisfaction in a Climate Science MOOC. The MOOC{\textquoteright}s learning assignments had learners choose resources on climate-related issues in either their geographic locale (Personalized group) or in given regions (Generic group). 219 learners completed at least one of the two assignments. Over the entire course, personalization increased learners{\textquoteright} activity (number of course events), self-reported understanding of local issues, and self-reported likelihood to change climate-related habits. We found no differences on assignment completion rate, assignment length, and self-reported time-on-task. These results show that benefits of personalization extend beyond the original task and affect learners{\textquoteright} overall experience. We discuss design and implications of choice-based personalization, as well as opportunities for choice-based personalization at scale.",
keywords = "Climate Science Education, Learning Design, MOOC, Personalization",
author = "Ilana Ram and Sara Harris and Ido Roll",
note = "Publisher Copyright: {\textcopyright} The Author(s) 2023.",
year = "2024",
month = jun,
doi = "10.1007/s40593-023-00334-5",
language = "אנגלית",
volume = "34",
pages = "376--394",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "2",

}

2023

Supporting students’ self-regulated learning in online learning using artificial intelligence applications

Jin SH, Im K, Yoo M, Roll I, Seo K. Supporting students’ self-regulated learning in online learning using artificial intelligence applications. International Journal of Educational Technology in Higher Education. 2023 Dec;20(1):37. https://doi.org/10.1186/s41239-023-00406-5
 

Self-regulated learning (SRL) is crucial for helping students attain high academic performance and achieve their learning objectives in the online learning context. However, learners often face challenges in properly applying SRL in online learning environments. Recent developments in artificial intelligence (AI) applications have shown promise in supporting learners’ self-regulation in online learning by measuring and augmenting SRL, but research in this area is still in its early stages. The purpose of this study is to explore students’ perceptions of the use of AI applications to support SRL and to identify the pedagogical and psychological aspects that they perceive as necessary for effective utilization of those AI applications. To explore this, a speed dating method using storyboards was employed as an exploratory design method. The study involved the development of 10 AI application storyboards to identify the phases and areas of SRL, and semi-structured interviews were conducted with 16 university students from various majors. The results indicated that learners perceived AI applications as useful for supporting metacognitive, cognitive, and behavioral regulation across different SRL areas, but not for regulating motivation. Next, regarding the use of AI applications to support SRL, learners requested consideration of three pedagogical and psychological aspects: learner identity, learner activeness, and learner position. The findings of this study offer practical implications for the design of AI applications in online learning, with the aim of supporting students’ SRL.

@article{7842e34cb3e74b4e881b77ba23e55fa6,
title = "Supporting students{\textquoteright} self-regulated learning in online learning using artificial intelligence applications",
abstract = "Self-regulated learning (SRL) is crucial for helping students attain high academic performance and achieve their learning objectives in the online learning context. However, learners often face challenges in properly applying SRL in online learning environments. Recent developments in artificial intelligence (AI) applications have shown promise in supporting learners{\textquoteright} self-regulation in online learning by measuring and augmenting SRL, but research in this area is still in its early stages. The purpose of this study is to explore students{\textquoteright} perceptions of the use of AI applications to support SRL and to identify the pedagogical and psychological aspects that they perceive as necessary for effective utilization of those AI applications. To explore this, a speed dating method using storyboards was employed as an exploratory design method. The study involved the development of 10 AI application storyboards to identify the phases and areas of SRL, and semi-structured interviews were conducted with 16 university students from various majors. The results indicated that learners perceived AI applications as useful for supporting metacognitive, cognitive, and behavioral regulation across different SRL areas, but not for regulating motivation. Next, regarding the use of AI applications to support SRL, learners requested consideration of three pedagogical and psychological aspects: learner identity, learner activeness, and learner position. The findings of this study offer practical implications for the design of AI applications in online learning, with the aim of supporting students{\textquoteright} SRL.",
keywords = "Artificial intelligence, Online learning, Self-regulated learning, Student perception",
author = "Jin, {Sung Hee} and Kowoon Im and Mina Yoo and Ido Roll and Kyoungwon Seo",
note = "Publisher Copyright: {\textcopyright} 2023, The Author(s).",
year = "2023",
month = dec,
doi = "10.1186/s41239-023-00406-5",
language = "אנגלית",
volume = "20",
journal = "International Journal of Educational Technology in Higher Education",
issn = "1698-580X",
publisher = "Springer Netherlands",
number = "1",

}

Prior math achievement and inventive production predict learning from productive failure

Kapur M, Saba J, Roll I. Prior math achievement and inventive production predict learning from productive failure. npj Science of Learning. 2023 Dec;8(1):15. https://doi.org/10.1038/s41539-023-00165-y
 

A frequent concern about constructivist instruction is that it works well, mainly for students with higher domain knowledge. We present findings from a set of two quasi-experimental pretest-intervention-posttest studies investigating the relationship between prior math achievement and learning in the context of a specific type of constructivist instruction, Productive Failure. Students from two Singapore public schools with significantly different prior math achievement profiles were asked to design solutions to complex problems prior to receiving instruction on the targeted concepts. Process results revealed that students who were significantly dissimilar in prior math achievement seemed to be strikingly similar in terms of their inventive production, that is, the variety of solutions they were able to design. Interestingly, it was inventive production that had a stronger association with learning from PF than pre-existing differences in math achievement. These findings, consistent across both topics, demonstrate the value of engaging students in opportunities for inventive production while learning math, regardless of prior math achievement.

@article{0d8887615d814ccb87d10bff69401d15,
title = "Prior math achievement and inventive production predict learning from productive failure",
abstract = "A frequent concern about constructivist instruction is that it works well, mainly for students with higher domain knowledge. We present findings from a set of two quasi-experimental pretest-intervention-posttest studies investigating the relationship between prior math achievement and learning in the context of a specific type of constructivist instruction, Productive Failure. Students from two Singapore public schools with significantly different prior math achievement profiles were asked to design solutions to complex problems prior to receiving instruction on the targeted concepts. Process results revealed that students who were significantly dissimilar in prior math achievement seemed to be strikingly similar in terms of their inventive production, that is, the variety of solutions they were able to design. Interestingly, it was inventive production that had a stronger association with learning from PF than pre-existing differences in math achievement. These findings, consistent across both topics, demonstrate the value of engaging students in opportunities for inventive production while learning math, regardless of prior math achievement.",
author = "Manu Kapur and Janan Saba and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2023, The Author(s).",
year = "2023",
month = dec,
doi = "10.1038/s41539-023-00165-y",
language = "אנגלית",
volume = "8",
journal = "npj Science of Learning",
issn = "2056-7936",
publisher = "Nature Publishing Group",
number = "1",

}

The Development of Multivariable Causality Strategy: Instruction or Simulation First?

Saba J, Kapur M, Roll I. The Development of Multivariable Causality Strategy: Instruction or Simulation First? In Wang N, Rebolledo-Mendez G, Matsuda N, Santos OC, Dimitrova V, editors, Artificial Intelligence in Education - 24th International Conference, AIED 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 41-53. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-031-36272-9_4
 

Understanding phenomena by exploring complex interactions between variables is a challenging task for students of all ages. While the use of simulations to support exploratory learning of complex phenomena is common, students still struggle to make sense of interactive relationships between factors. Here we study the applicability of Problem Solving before Instruction (PS-I) approach to this context. In PS-I, learners are given complex tasks that help them make sense of the domain, prior to receiving instruction on the target concepts. While PS-I has been shown to be effective to teach complex topics, it is yet to show benefits for learning general inquiry skills. Thus, we tested the effect of exploring with simulations before instruction (as opposed to afterward) on the development of a multivariable causality strategy (MVC-strategy). Undergraduate students (N = 71) completed two exploration tasks using simulation about virus transmission. Students completed Task1 either before (Exploration-first condition) or after (Instruction-first condition) instruction related to multivariable causality and completed Task2 at the end of the intervention. Following, they completed transfer Task3 with a simulation on the topic of Predator-Prey relationships. Results showed that Instruction-first improved students’ Efficiency of MVC-strategy on Task1. However, these gaps were gone by Task2. Interestingly, Exploration-first had higher efficiency of MVC-strategy on transfer Task3. These results show that while Exploration-first did not promote performance on the learning activity, it has in fact improved learning on the transfer task, consistent with the PS-I literature. This is the first time that PS-I is found effective in teaching students better exploration strategies.

@inproceedings{e75df67034a5426ab7f2b690c84f3888,
title = "The Development of Multivariable Causality Strategy: Instruction or Simulation First?",
abstract = "Understanding phenomena by exploring complex interactions between variables is a challenging task for students of all ages. While the use of simulations to support exploratory learning of complex phenomena is common, students still struggle to make sense of interactive relationships between factors. Here we study the applicability of Problem Solving before Instruction (PS-I) approach to this context. In PS-I, learners are given complex tasks that help them make sense of the domain, prior to receiving instruction on the target concepts. While PS-I has been shown to be effective to teach complex topics, it is yet to show benefits for learning general inquiry skills. Thus, we tested the effect of exploring with simulations before instruction (as opposed to afterward) on the development of a multivariable causality strategy (MVC-strategy). Undergraduate students (N = 71) completed two exploration tasks using simulation about virus transmission. Students completed Task1 either before (Exploration-first condition) or after (Instruction-first condition) instruction related to multivariable causality and completed Task2 at the end of the intervention. Following, they completed transfer Task3 with a simulation on the topic of Predator-Prey relationships. Results showed that Instruction-first improved students{\textquoteright} Efficiency of MVC-strategy on Task1. However, these gaps were gone by Task2. Interestingly, Exploration-first had higher efficiency of MVC-strategy on transfer Task3. These results show that while Exploration-first did not promote performance on the learning activity, it has in fact improved learning on the transfer task, consistent with the PS-I literature. This is the first time that PS-I is found effective in teaching students better exploration strategies.",
keywords = "Interactive simulation, Multivariable causality strategy, exploratory learning",
author = "Janan Saba and Manu Kapur and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 24th International Conference on Artificial Intelligence in Education, AIED 2023 ; Conference date: 03-07-2023 Through 07-07-2023",
year = "2023",
doi = "10.1007/978-3-031-36272-9_4",
language = "אנגלית",
isbn = "9783031362712",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "41--53",
editor = "Ning Wang and Genaro Rebolledo-Mendez and Noboru Matsuda and Santos, {Olga C.} and Vania Dimitrova",
booktitle = "Artificial Intelligence in Education - 24th International Conference, AIED 2023, Proceedings",
address = "גרמניה",

}

Desirable Difficulties? The Effects of Spaced and Interleaved Practice in an Educational Game

Ben-David J, Roll I. Desirable Difficulties? The Effects of Spaced and Interleaved Practice in an Educational Game. In Wang N, Rebolledo-Mendez G, Dimitrova V, Matsuda N, Santos OC, editors, Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 136-141. (Communications in Computer and Information Science). https://doi.org/10.1007/978-3-031-36336-8_21
 

Educational games benefit from incorporating evidence-based learning principles. Most of these principles have been studied in conventional learning settings, and their applicability and possible benefits to games is unclear. Two such principles within the general framework of Desirable Difficulties are Spacing and Interleaving. Both refer to the idea of distributed practice over time (as opposed to blocked), with interleaving having the added benefit of presenting opportunities for comparison across stimuli. We designed a digital game on the topic of multiplication facts that implemented three possible sequencing of problems – blocked, spaced, and interleaved. One hundred and fifty elementary school students were randomly assigned to one of the conditions. In-game learning curve analysis of logs found that blocked presentation improves in-game performance. However, out-of-game tests found that interleaved presentation improves out-of-game performance efficiency. This work demonstrates the applicability and benefits of incorporating evidence-based principles into digital games.

@inproceedings{0e41af94945a43f7b2eb9a5eac1acd1b,
title = "Desirable Difficulties? The Effects of Spaced and Interleaved Practice in an Educational Game",
abstract = "Educational games benefit from incorporating evidence-based learning principles. Most of these principles have been studied in conventional learning settings, and their applicability and possible benefits to games is unclear. Two such principles within the general framework of Desirable Difficulties are Spacing and Interleaving. Both refer to the idea of distributed practice over time (as opposed to blocked), with interleaving having the added benefit of presenting opportunities for comparison across stimuli. We designed a digital game on the topic of multiplication facts that implemented three possible sequencing of problems – blocked, spaced, and interleaved. One hundred and fifty elementary school students were randomly assigned to one of the conditions. In-game learning curve analysis of logs found that blocked presentation improves in-game performance. However, out-of-game tests found that interleaved presentation improves out-of-game performance efficiency. This work demonstrates the applicability and benefits of incorporating evidence-based principles into digital games.",
keywords = "Classroom experiment, Desirable difficulties, Educational games, Learning rates, Multiplication facts",
author = "Jonathan Ben-David and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 24th International Conference on Artificial Intelligence in Education , AIED 2023 ; Conference date: 03-07-2023 Through 07-07-2023",
year = "2023",
doi = "10.1007/978-3-031-36336-8_21",
language = "אנגלית",
isbn = "9783031363351",
series = "Communications in Computer and Information Science",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "136--141",
editor = "Ning Wang and Genaro Rebolledo-Mendez and Vania Dimitrova and Noboru Matsuda and Santos, {Olga C.}",
booktitle = "Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings",
address = "גרמניה",

}

Consistency of Inquiry Strategies Across Subsequent Activities in Different Domains

Cock JM, Roll I, Käser T. Consistency of Inquiry Strategies Across Subsequent Activities in Different Domains. In Wang N, Rebolledo-Mendez G, Dimitrova V, Matsuda N, Santos OC, editors, Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 224-229. (Communications in Computer and Information Science). https://doi.org/10.1007/978-3-031-36336-8_34
 

Interactive simulations encourage students to practice skills essential to understanding and learning sciences. Alas, inquiry learning with interactive simulations is challenging. In this paper, we seek to identify inquiry patterns across topics and evaluate their stability with regard to common behaviors and student membership. Applying a clustering approach, we propose an encoding through which we can model students’ strategies in diverse environments. Specifically, we encode each sequence with three different levels of granularity which range from simulation-specific characteristics to simulation-agnostic features. Using this generalizable encoding, we find two clusters for each of two simulations. The formed groups exhibit similar learning patterns across environments. One systematically cycles through exploring and recording systematically over all variables. The other group explores the simulation more freely. This suggests that our feature encoding captures inherent quality of inquiry with simulations and can be used to characterize learners knowledge of productive exploration.

@inproceedings{c77d3a11d7ee49c888e3da9c7d1c901c,
title = "Consistency of Inquiry Strategies Across Subsequent Activities in Different Domains",
abstract = "Interactive simulations encourage students to practice skills essential to understanding and learning sciences. Alas, inquiry learning with interactive simulations is challenging. In this paper, we seek to identify inquiry patterns across topics and evaluate their stability with regard to common behaviors and student membership. Applying a clustering approach, we propose an encoding through which we can model students{\textquoteright} strategies in diverse environments. Specifically, we encode each sequence with three different levels of granularity which range from simulation-specific characteristics to simulation-agnostic features. Using this generalizable encoding, we find two clusters for each of two simulations. The formed groups exhibit similar learning patterns across environments. One systematically cycles through exploring and recording systematically over all variables. The other group explores the simulation more freely. This suggests that our feature encoding captures inherent quality of inquiry with simulations and can be used to characterize learners knowledge of productive exploration.",
keywords = "inquiry skills, inquiry strategies, interactive simulation, log data, open ended learning environments, spectral clustering",
author = "Cock, {Jade Mai} and Ido Roll and Tanja K{\"a}ser",
note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 24th International Conference on Artificial Intelligence in Education , AIED 2023 ; Conference date: 03-07-2023 Through 07-07-2023",
year = "2023",
doi = "10.1007/978-3-031-36336-8_34",
language = "אנגלית",
isbn = "9783031363351",
series = "Communications in Computer and Information Science",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "224--229",
editor = "Ning Wang and Genaro Rebolledo-Mendez and Vania Dimitrova and Noboru Matsuda and Santos, {Olga C.}",
booktitle = "Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings",
address = "גרמניה",

}

2022

Rethinking (Dis)engagement in human-computer interaction

O'Brien HL, Roll I, Kampen A, Davoudi N. Rethinking (Dis)engagement in human-computer interaction. Computers in Human Behavior. 2022 Mar 1;128:107109. https://doi.org/10.1016/j.chb.2021.107109
 

User engagement has become a much-cited construct in human-computer interaction (HCI) design and evaluation research and practice. Constructed as a positive and desirable outcome of users' interactions, more frequent and longer interactions are considered evidence of engagement. Disengagement, when discussed, is considered a best avoided outcome of technology use or a solution to problematic technology use. In the case of the former, disengagement may signal usability issues or user disinterest, while the latter emphasizes that some engaging interactions can result in negative consequences (e.g., addiction) for end-users. In this paper, we draw upon examples from HCI research and digital tools to present a more nuanced understanding of the symbiotic relationship between engagement and disengagement in order to propose a new definition and novel ways to model disengagement. Further, we challenge generalizations that dichotomize engagement (positive, continuous, accompanied by high interactivity and beneficial to end-users) and disengagement (negative, stopping use or detrimental use) and invite readers to interpret engagement in the context of desirability with respect to users' goals and perceived agency. We concluded with implications that invite the reader to make space for disengagement and move beyond usage data in the evaluation of engagement. This paper is a call to step away from the practice of engagement-for-engagement's sake, and to reflect on whether and when engagement is meaningful and desirable for end users.

@article{9e33ec3b7137426abf6e30b346662489,
title = "Rethinking (Dis)engagement in human-computer interaction",
abstract = "User engagement has become a much-cited construct in human-computer interaction (HCI) design and evaluation research and practice. Constructed as a positive and desirable outcome of users' interactions, more frequent and longer interactions are considered evidence of engagement. Disengagement, when discussed, is considered a best avoided outcome of technology use or a solution to problematic technology use. In the case of the former, disengagement may signal usability issues or user disinterest, while the latter emphasizes that some engaging interactions can result in negative consequences (e.g., addiction) for end-users. In this paper, we draw upon examples from HCI research and digital tools to present a more nuanced understanding of the symbiotic relationship between engagement and disengagement in order to propose a new definition and novel ways to model disengagement. Further, we challenge generalizations that dichotomize engagement (positive, continuous, accompanied by high interactivity and beneficial to end-users) and disengagement (negative, stopping use or detrimental use) and invite readers to interpret engagement in the context of desirability with respect to users' goals and perceived agency. We concluded with implications that invite the reader to make space for disengagement and move beyond usage data in the evaluation of engagement. This paper is a call to step away from the practice of engagement-for-engagement's sake, and to reflect on whether and when engagement is meaningful and desirable for end users.",
keywords = "Disengagement, Human-computer interaction, User engagement, User experience design and evaluation",
author = "O'Brien, {Heather L.} and Ido Roll and Andrea Kampen and Nilou Davoudi",
note = "Publisher Copyright: {\textcopyright} 2021 The Authors",
year = "2022",
month = mar,
day = "1",
doi = "10.1016/j.chb.2021.107109",
language = "אנגלית",
volume = "128",
journal = "Computers in Human Behavior",
issn = "0747-5632",
publisher = "Elsevier Ltd.",

}

The effect of different sequences of examples and problems on learning experimental design

Ganaiem E, Roll I. The effect of different sequences of examples and problems on learning experimental design. In Chinn C, Tan E, Chan C, Kali Y, editors, International Collaboration toward Educational Innovation for All: Overarching Research, Development, and Practices - 16th International Conference of the Learning Sciences, ICLS 2022. International Society of the Learning Sciences (ISLS). 2022. p. 727-734. (Proceedings of International Conference of the Learning Sciences, ICLS).
 

Example-based learning refers to a pedagogy in which learners are provided with a step-by-step solution to novel problems early in the learning process. Worked examples are most effective when used in tandem with open problems. However, there are still questions regarding the ordering of examples and problems. We compared the effect of sequencing examples and problems on learning and cognitive load. One hundred thirteen middle-school students were randomly assigned to one of three conditions that varied presentation order within each example-problem pair: example first, problem first, or simultaneous presentation. We evaluated students' performances on knowledge application and the transfer of learning on the topic of experimental design. We found no effect neither on learning nor on cognitive load. We compare these results to parallel studies in the literature and identify possible reasons for the null effect.

@inproceedings{ba25ad91857c4cdea96692f1e2a4c973,
title = "The effect of different sequences of examples and problems on learning experimental design",
abstract = "Example-based learning refers to a pedagogy in which learners are provided with a step-by-step solution to novel problems early in the learning process. Worked examples are most effective when used in tandem with open problems. However, there are still questions regarding the ordering of examples and problems. We compared the effect of sequencing examples and problems on learning and cognitive load. One hundred thirteen middle-school students were randomly assigned to one of three conditions that varied presentation order within each example-problem pair: example first, problem first, or simultaneous presentation. We evaluated students' performances on knowledge application and the transfer of learning on the topic of experimental design. We found no effect neither on learning nor on cognitive load. We compare these results to parallel studies in the literature and identify possible reasons for the null effect.",
author = "Eman Ganaiem and Ido Roll",
note = "Publisher Copyright: {\textcopyright} ISLS.; 16th International Conference of the Learning Sciences, ICLS 2022 ; Conference date: 06-06-2022 Through 10-06-2022",
year = "2022",
language = "אנגלית",
series = "Proceedings of International Conference of the Learning Sciences, ICLS",
publisher = "International Society of the Learning Sciences (ISLS)",
pages = "727--734",
editor = "Clark Chinn and Edna Tan and Carol Chan and Yael Kali",
booktitle = "International Collaboration toward Educational Innovation for All",

}

הכשרת מנהיגות העתיד

צ'ונטונוב א, רול ע. הכשרת מנהיגות העתיד. הטכניון. 2022;8-9.
 
שיפור ההוראה ושכלול דרכי ההערכה הם כיום משימות מרכזיות בסדר היום הטכניוני. (מתוך המאמר)
@article{12e150114cb44471b86712369ecbc3ac,
title = "הכשרת מנהיגות העתיד",
abstract = "שיפור ההוראה ושכלול דרכי ההערכה הם כיום משימות מרכזיות בסדר היום הטכניוני. (מתוך המאמר)",
author = "אולגה צ'ונטונוב and עידו רול",
note = "ד{"}ר אולגה צ'ונטונוב ופרופ' עדו רול.",
year = "2022",
language = "עברית",
pages = "8--9",
journal = "הטכניון",
issn = "0793-8543",

}

Faculty and Student Partnerships in the Scholarship of Teaching and Learning: Evaluation of an Institutional Model

Moghtader B, Briseño-Garzón A, Varao-Sousa T, Roll I. Faculty and Student Partnerships in the Scholarship of Teaching and Learning: Evaluation of an Institutional Model. Teaching and Learning Inquiry. 2022;10. https://doi.org/10.20343/TEACHLEARNINQU.10.33
 

We present the design and evaluation of an institutional support model for the Scholarship of Teaching and Learning (SoTL): the SoTL Seed Program. In this model, faculty from across disciplines partner with graduate students with expertise in educational and social science methodologies to implement SoTL investigations. We interviewed and obtained feedback from both faculty and graduate students about their experiences. A qualitative approach based on grounded theory suggests that organized and sustained partnership between faculty and graduate students offers a viable institutional framework to support SoTL across academic disciplines. In our institution, partnerships in SoTL have resulted in facilitating academic and professional development for both faculty and graduate students, establishing communities of practice for SoTL, and providing infrastructure for systematic engagement with SoTL.

@article{75d12feadf814f2ab336e6dfb15a9e5c,
title = "Faculty and Student Partnerships in the Scholarship of Teaching and Learning: Evaluation of an Institutional Model",
abstract = "We present the design and evaluation of an institutional support model for the Scholarship of Teaching and Learning (SoTL): the SoTL Seed Program. In this model, faculty from across disciplines partner with graduate students with expertise in educational and social science methodologies to implement SoTL investigations. We interviewed and obtained feedback from both faculty and graduate students about their experiences. A qualitative approach based on grounded theory suggests that organized and sustained partnership between faculty and graduate students offers a viable institutional framework to support SoTL across academic disciplines. In our institution, partnerships in SoTL have resulted in facilitating academic and professional development for both faculty and graduate students, establishing communities of practice for SoTL, and providing infrastructure for systematic engagement with SoTL.",
keywords = "academic, community of practice, faculty and student partnership, institutional support, professional development",
author = "Bruce Moghtader and Adriana Brise{\~n}o-Garz{\'o}n and Trish Varao-Sousa and Ido Roll",
note = "Publisher Copyright: Copyright for the content of articles published in Teaching & Learning Inquiry resides with the authors, and copyright for the publication layout resides with the journal.",
year = "2022",
doi = "10.20343/TEACHLEARNINQU.10.33",
language = "אנגלית",
volume = "10",
journal = "Teaching and Learning Inquiry",
issn = "2167-4787",
publisher = "University of Calgary Press",

}

2021

The impact of artificial intelligence on learner–instructor interaction in online learning

Seo K, Tang J, Roll I, Fels S, Yoon D. The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education. 2021 Dec;18(1). https://doi.org/10.1186/s41239-021-00292-9
 

Artificial intelligence (AI) systems offer effective support for online learning and teaching, including personalizing learning for students, automating instructors’ routine tasks, and powering adaptive assessments. However, while the opportunities for AI are promising, the impact of AI systems on the culture of, norms in, and expectations about interactions between students and instructors are still elusive. In online learning, learner–instructor interaction (inter alia, communication, support, and presence) has a profound impact on students’ satisfaction and learning outcomes. Thus, identifying how students and instructors perceive the impact of AI systems on their interaction is important to identify any gaps, challenges, or barriers preventing AI systems from achieving their intended potential and risking the safety of these interactions. To address this need for forward-looking decisions, we used Speed Dating with storyboards to analyze the authentic voices of 12 students and 11 instructors on diverse use cases of possible AI systems in online learning. Findings show that participants envision adopting AI systems in online learning can enable personalized learner–instructor interaction at scale but at the risk of violating social boundaries. Although AI systems have been positively recognized for improving the quantity and quality of communication, for providing just-in-time, personalized support for large-scale settings, and for improving the feeling of connection, there were concerns about responsibility, agency, and surveillance issues. These findings have implications for the design of AI systems to ensure explainability, human-in-the-loop, and careful data collection and presentation. Overall, contributions of this study include the design of AI system storyboards which are technically feasible and positively support learner–instructor interaction, capturing students’ and instructors’ concerns of AI systems through Speed Dating, and suggesting practical implications for maximizing the positive impact of AI systems while minimizing the negative ones.

@article{e42c10dc7aab47a6b307989f97b2368c,
title = "The impact of artificial intelligence on learner–instructor interaction in online learning",
abstract = "Artificial intelligence (AI) systems offer effective support for online learning and teaching, including personalizing learning for students, automating instructors{\textquoteright} routine tasks, and powering adaptive assessments. However, while the opportunities for AI are promising, the impact of AI systems on the culture of, norms in, and expectations about interactions between students and instructors are still elusive. In online learning, learner–instructor interaction (inter alia, communication, support, and presence) has a profound impact on students{\textquoteright} satisfaction and learning outcomes. Thus, identifying how students and instructors perceive the impact of AI systems on their interaction is important to identify any gaps, challenges, or barriers preventing AI systems from achieving their intended potential and risking the safety of these interactions. To address this need for forward-looking decisions, we used Speed Dating with storyboards to analyze the authentic voices of 12 students and 11 instructors on diverse use cases of possible AI systems in online learning. Findings show that participants envision adopting AI systems in online learning can enable personalized learner–instructor interaction at scale but at the risk of violating social boundaries. Although AI systems have been positively recognized for improving the quantity and quality of communication, for providing just-in-time, personalized support for large-scale settings, and for improving the feeling of connection, there were concerns about responsibility, agency, and surveillance issues. These findings have implications for the design of AI systems to ensure explainability, human-in-the-loop, and careful data collection and presentation. Overall, contributions of this study include the design of AI system storyboards which are technically feasible and positively support learner–instructor interaction, capturing students{\textquoteright} and instructors{\textquoteright} concerns of AI systems through Speed Dating, and suggesting practical implications for maximizing the positive impact of AI systems while minimizing the negative ones.",
keywords = "Artificial intelligence, Boundary, Learner–instructor interaction, Online learning, Speed dating",
author = "Kyoungwon Seo and Joice Tang and Ido Roll and Sidney Fels and Dongwook Yoon",
note = "Publisher Copyright: {\textcopyright} 2021, The Author(s).",
year = "2021",
month = dec,
doi = "10.1186/s41239-021-00292-9",
language = "אנגלית",
volume = "18",
journal = "International Journal of Educational Technology in Higher Education",
issn = "1698-580X",
publisher = "Springer Netherlands",
number = "1",

}

Personalization at Scale: Making Learning Personally Relevant in a Climate Science MOOC

Roll I, Ram I, Harris S. Personalization at Scale: Making Learning Personally Relevant in a Climate Science MOOC. In L@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale. Association for Computing Machinery, Inc. 2021. p. 263-266. (Proceedings of the Eighth ACM Conference on Learning @ Scale). https://doi.org/10.1145/3430895.3460154
 

Personalization and choice in learning activities can increase student engagement, satisfaction, and learning gains. But does this effect hold when implemented at scale? The current work explores the effects of personalization and learner's choice in a Climate Science MOOC. We manipulated these by creating two versions of course assignments. Learners who completed the assignments (N=219) received either Generic assignments focusing on global climate issues or Personalized assignments in which learners explored their own regions. Following the manipulation, learners in the Personalization group reported equal understanding of both Global and Local climate issues while learners in the Generic group reported better understanding of global issues and reduced understanding of local issues. Further, personalization did not affect interest or assignment length. We describe opportunities for personalization at scale and discuss their outcomes.

@inproceedings{3ac7a8f9184d4b3aa431e43ea60edca7,
title = "Personalization at Scale: Making Learning Personally Relevant in a Climate Science MOOC",
abstract = "Personalization and choice in learning activities can increase student engagement, satisfaction, and learning gains. But does this effect hold when implemented at scale? The current work explores the effects of personalization and learner's choice in a Climate Science MOOC. We manipulated these by creating two versions of course assignments. Learners who completed the assignments (N=219) received either Generic assignments focusing on global climate issues or Personalized assignments in which learners explored their own regions. Following the manipulation, learners in the Personalization group reported equal understanding of both Global and Local climate issues while learners in the Generic group reported better understanding of global issues and reduced understanding of local issues. Further, personalization did not affect interest or assignment length. We describe opportunities for personalization at scale and discuss their outcomes.",
keywords = "learning design, mooc, personalization, value",
author = "Ido Roll and Ilana Ram and Sara Harris",
note = "Publisher Copyright: {\textcopyright} 2021 Owner/Author.; 8th Annual ACM Conference on Learning at Scale, L@S 2021 ; Conference date: 22-06-2021 Through 25-06-2021",
year = "2021",
month = jun,
day = "8",
doi = "10.1145/3430895.3460154",
language = "אנגלית",
series = "Proceedings of the Eighth ACM Conference on Learning @ Scale",
publisher = "Association for Computing Machinery, Inc",
pages = "263--266",
booktitle = "L@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale",

}

Active learning with online video: The impact of learning context on engagement

Seo K, Dodson S, Harandi NM, Roberson N, Fels S, Roll I. Active learning with online video: The impact of learning context on engagement. Computers and Education. 2021 May;165:104132. https://doi.org/10.1016/j.compedu.2021.104132
 

Learning with online video is pervasive in higher education. Recent research has explored the importance of student engagement when learning with video in online and blended courses. However, little is known about students' goals and intents when engaging with video. Furthermore, there is limited empirical evidence on the impact of learning context on engagement with video, which limits our understanding of how students learn from video. To address this gap, we identify a set of engagement goals for learning with video, and study associated student activity in relation to learning context (course week, exam, and rewatch). In Study 1, we conducted a survey (n = 116) that maps students' video viewing activities to their engagement goals and intents. We identified a variety of engagement goals, specifically Reflect, Flag, Remember, Clarify, Skim, Search, Orient, and Take a break. In Study 2, we analyzed clickstream data generated by 387 students enrolled in three semester-long courses. We examined the impact of learning context on students’ engagement with video. A multilevel model showed different patterns for online and blended courses. Students in the online course showed much more strategic and adaptive use of video. As the semester progressed, students in the online courses performed fewer Reflect and Search. During exam weeks and when rewatching videos, online students performed more Search within the video. The only trend that was found for blended learning students was an increase in Skim with course week. These findings have implications for video players that adapt to context, such as helping students easily locate important in-video information during the exam week or when rewatching previously watched videos.

@article{0bc9880a013845a0b5bcec7e787f6d05,
title = "Active learning with online video: The impact of learning context on engagement",
abstract = "Learning with online video is pervasive in higher education. Recent research has explored the importance of student engagement when learning with video in online and blended courses. However, little is known about students' goals and intents when engaging with video. Furthermore, there is limited empirical evidence on the impact of learning context on engagement with video, which limits our understanding of how students learn from video. To address this gap, we identify a set of engagement goals for learning with video, and study associated student activity in relation to learning context (course week, exam, and rewatch). In Study 1, we conducted a survey (n = 116) that maps students' video viewing activities to their engagement goals and intents. We identified a variety of engagement goals, specifically Reflect, Flag, Remember, Clarify, Skim, Search, Orient, and Take a break. In Study 2, we analyzed clickstream data generated by 387 students enrolled in three semester-long courses. We examined the impact of learning context on students{\textquoteright} engagement with video. A multilevel model showed different patterns for online and blended courses. Students in the online course showed much more strategic and adaptive use of video. As the semester progressed, students in the online courses performed fewer Reflect and Search. During exam weeks and when rewatching videos, online students performed more Search within the video. The only trend that was found for blended learning students was an increase in Skim with course week. These findings have implications for video players that adapt to context, such as helping students easily locate important in-video information during the exam week or when rewatching previously watched videos.",
keywords = "Engagement, Learning analytics, Learning context, Online/blended courses, Video-based learning",
author = "Kyoungwon Seo and Samuel Dodson and Harandi, {Negar M.} and Nathan Roberson and Sidney Fels and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2021 Elsevier Ltd",
year = "2021",
month = may,
doi = "10.1016/j.compedu.2021.104132",
language = "אנגלית",
volume = "165",
journal = "Computers and Education",
issn = "0360-1315",
publisher = "Elsevier Ltd.",

}

Interface and interaction design for an online, asynchronous peer instruction tool

Englund L, Moosvi F, Roll I. Interface and interaction design for an online, asynchronous peer instruction tool. Interactive Learning Environments. 2021 Apr 14;31(5):2809-2829. https://doi.org/10.1080/10494820.2021.1910849
 

This article describes the design and evaluation of an online, asynchronous tool that mirrors the beneficial peer instruction process currently conducted in face-to-face classrooms. In the online Peer Instruction tool, students respond to a multiple-choice question with an answer and explanation. They are then exposed to peer responses to the same question. Finally, they may optionally submit a revised response, formed after exposure to these peer responses. Results from an experiment in three large-scale courses highlighted the perceived benefits for students who chose to engage meaningfully with the tool and showed opportunities for improving the user experience to increase engagement. After redesigning the interface and interaction, the original and revised versions of the tool were assessed in usability testing with students. The findings suggested the redesign is an improvement, reveal potential reasons the original design may have failed to engage more students, and reinforce the importance of understanding the multiple layers of the student experience when designing any educational technology.

@article{6b3d000a537f4700a1292f2d4ad770a1,
title = "Interface and interaction design for an online, asynchronous peer instruction tool",
abstract = "This article describes the design and evaluation of an online, asynchronous tool that mirrors the beneficial peer instruction process currently conducted in face-to-face classrooms. In the online Peer Instruction tool, students respond to a multiple-choice question with an answer and explanation. They are then exposed to peer responses to the same question. Finally, they may optionally submit a revised response, formed after exposure to these peer responses. Results from an experiment in three large-scale courses highlighted the perceived benefits for students who chose to engage meaningfully with the tool and showed opportunities for improving the user experience to increase engagement. After redesigning the interface and interaction, the original and revised versions of the tool were assessed in usability testing with students. The findings suggested the redesign is an improvement, reveal potential reasons the original design may have failed to engage more students, and reinforce the importance of understanding the multiple layers of the student experience when designing any educational technology.",
keywords = "Iterative design, online engagement, peer instruction, student experience, usability",
author = "L. Englund and F. Moosvi and I. Roll",
note = "Publisher Copyright: {\textcopyright} 2021 Informa UK Limited, trading as Taylor & Francis Group.",
year = "2021",
month = apr,
day = "14",
doi = "10.1080/10494820.2021.1910849",
language = "אנגלית",
volume = "31",
pages = "2809--2829",
journal = "Interactive Learning Environments",
issn = "1049-4820",
publisher = "Taylor and Francis Ltd.",
number = "5",

}

Towards Asynchronous Data Science Invention Activities at Scale

Shalala R, Amir O, Roll I. Towards Asynchronous Data Science Invention Activities at Scale. In Hmelo-Silver CE, De Wever B, Oshima J, editors, 14th International Conference on Computer-Supported Collaborative Learning: Reflecting the Past and Embracing the Future, CSCL 2021 - Proceedings, part of the 1st Annual Meeting of the International Society of the Learning Sciences, ISLS 2021. International Society of the Learning Sciences (ISLS). 2021. p. 43-50. (Computer-Supported Collaborative Learning Conference, CSCL).
 

Invention activities are carefully designed problem-solving tasks in which learners are asked to invent solutions to unfamiliar problems prior to being taught the canonical solutions. Invention activities are typically used in the classroom setting. As online education becomes increasingly common, there is a need to adapt Invention activities to the asynchronous nature of many courses. We do so in the context of an introductory undergraduate data science course. Using an online programming environment, students work on the tasks in pairs, without instructor support. We analyze the invention process and outcomes from two Invention activities on the challenging topics of classification and clustering. Detailed analysis of recordings of six student pairs shows how activity design supports insights at three levels: nature of models (e.g., the need to normalize); domain concepts (e.g., types of errors), and procedural solutions (e.g., weighting errors). We describe the activities, their design, and their outcomes.

@inproceedings{def1eea8ca8f4788b565fad68f56d35c,
title = "Towards Asynchronous Data Science Invention Activities at Scale",
abstract = "Invention activities are carefully designed problem-solving tasks in which learners are asked to invent solutions to unfamiliar problems prior to being taught the canonical solutions. Invention activities are typically used in the classroom setting. As online education becomes increasingly common, there is a need to adapt Invention activities to the asynchronous nature of many courses. We do so in the context of an introductory undergraduate data science course. Using an online programming environment, students work on the tasks in pairs, without instructor support. We analyze the invention process and outcomes from two Invention activities on the challenging topics of classification and clustering. Detailed analysis of recordings of six student pairs shows how activity design supports insights at three levels: nature of models (e.g., the need to normalize); domain concepts (e.g., types of errors), and procedural solutions (e.g., weighting errors). We describe the activities, their design, and their outcomes.",
author = "Rafael Shalala and Ofra Amir and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2021 International Society of the Learning Sciences (ISLS). All rights reserved.; 14th International Conference on Computer-Supported Collaborative Learning, CSCL 2021 ; Conference date: 08-06-2021 Through 11-06-2021",
year = "2021",
language = "אנגלית",
series = "Computer-Supported Collaborative Learning Conference, CSCL",
publisher = "International Society of the Learning Sciences (ISLS)",
pages = "43--50",
editor = "Hmelo-Silver, {Cindy E.} and {De Wever}, Bram and Jun Oshima",
booktitle = "14th International Conference on Computer-Supported Collaborative Learning",

}

Preface

Roll I, McNamara D, Sosnovsky S, Luckin R, Dimitrova V. Preface. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2021;12748 LNAI:v-vi. https://doi.org/10.1016/S0074-6142(08)60615-4
@article{c568e51c52964674b63f7c171c28c40f,
title = "Preface",
author = "Ido Roll and Danielle McNamara and Sergey Sosnovsky and Rose Luckin and Vania Dimitrova",
note = "Funding Information: – A Student Forum, funded by the Schmidt Foundation, that supported undergraduate students in learning about AIED, its past, present, and future challenges, and helped them make connections within the community. Special thanks go to Springer for sponsoring the AIED 2020 Best Paper Award. We also wish to acknowledge the wonderful work of the AIED 2020 Organizing Committee, the PC members, and the reviewers who made this conference possible. This conference was certainly a community effort and a testament to the community{\textquoteright}s strength.; 22nd International Conference on Artificial Intelligence in Education, AIED 2021 ; Conference date: 14-06-2021 Through 18-06-2021",
year = "2021",
doi = "10.1016/S0074-6142(08)60615-4",
language = "אנגלית",
volume = "12748 LNAI",
pages = "v--vi",
journal = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
issn = "0302-9743",
publisher = "Springer Verlag",

}

Preface

Roll I, McNamara D, Sosnovsky S, Luckin R, Dimitrova V. Preface. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2021;12749 LNAI:v-vi.
@article{cbc2cc25c3064f0286381f1456b1f5dc,
title = "Preface",
author = "Ido Roll and Danielle McNamara and Sergey Sosnovsky and Rose Luckin and Vania Dimitrova",
note = "Funding Information: – A Student Forum, funded by the Schmidt Foundation, that supported undergraduate students in learning about AIED, its past, present, and future challenges, and helped them make connections within the community. Special thanks go to Springer for sponsoring the AIED 2020 Best Paper Award. We also wish to acknowledge the wonderful work of the AIED 2020 Organizing Committee, the PC members, and the reviewers who made this conference possible. This conference was certainly a community effort and a testament to the community{\textquoteright}s strength.; 22nd International Conference on Artificial Intelligence in Education, AIED 2021 ; Conference date: 14-06-2021 Through 18-06-2021",
year = "2021",
language = "אנגלית",
volume = "12749 LNAI",
pages = "v--vi",
journal = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
issn = "0302-9743",
publisher = "Springer Verlag",

}

Towards Asynchronous Data Science Invention Activities at Scale

Shalala R, Amir O, Roll I. Towards Asynchronous Data Science Invention Activities at Scale. In International Society of the Learning Sciences (ISLS). 2021
@inproceedings{811efc40fa2045fab19df4ab678efc98,
title = "Towards Asynchronous Data Science Invention Activities at Scale",
author = "Rafi Shalala and Ofra Amir and Ido Roll",
year = "2021",
language = "American English",
booktitle = "International Society of the Learning Sciences (ISLS)",

}

2020

Artificial Intelligence for Video-based Learning at Scale

Seo K, Fels S, Yoon D, Roll I, Dodson S, Fong M. Artificial Intelligence for Video-based Learning at Scale. In L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale. Association for Computing Machinery. 2020. p. 215-217. (Proceedings of the Seventh ACM Conference on Learning @ Scale). https://doi.org/10.1145/3386527.3405937
 

Video-based learning (VBL) is widespread; however, there are numerous challenges when teaching and learning with video. For instructors, creating effective instructional videos takes considerable time and effort. For students, watching videos can be a passive learning activity. Artificial intelligence (AI) has the potential to improve the VBL experience for students and teachers. This half-day workshop will bring together multi-disciplinary researchers and practitioners to collaboratively envision the future of VBL enhanced by AI. This workshop will be comprised of a group discussion followed by a presentation session. The goal of the workshop is to facilitate the cross-pollination of design ideas and critical assessments of AI approaches to VBL.

@inproceedings{1e59907fde2c48d8bf18dd01d849d9d2,
title = "Artificial Intelligence for Video-based Learning at Scale",
abstract = "Video-based learning (VBL) is widespread; however, there are numerous challenges when teaching and learning with video. For instructors, creating effective instructional videos takes considerable time and effort. For students, watching videos can be a passive learning activity. Artificial intelligence (AI) has the potential to improve the VBL experience for students and teachers. This half-day workshop will bring together multi-disciplinary researchers and practitioners to collaboratively envision the future of VBL enhanced by AI. This workshop will be comprised of a group discussion followed by a presentation session. The goal of the workshop is to facilitate the cross-pollination of design ideas and critical assessments of AI approaches to VBL.",
keywords = "artificial intelligence, computer vision, machine learning, natural language processing, video-based learning",
author = "Kyoungwon Seo and Sidney Fels and Dongwook Yoon and Ido Roll and Samuel Dodson and Matthew Fong",
note = "Publisher Copyright: {\textcopyright} 2020 ACM.; 7th Annual ACM Conference on Learning at Scale, L@S 2020 ; Conference date: 12-08-2020 Through 14-08-2020",
year = "2020",
month = aug,
day = "12",
doi = "10.1145/3386527.3405937",
language = "אנגלית",
series = "Proceedings of the Seventh ACM Conference on Learning @ Scale",
publisher = "Association for Computing Machinery",
pages = "215--217",
booktitle = "L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale",

}

2019

DIY productive failure: boosting performance in a large undergraduate biology course

Chowrira SG, Smith KM, Dubois PJ, Roll I. DIY productive failure: boosting performance in a large undergraduate biology course. npj Science of Learning. 2019 Dec;4(1):1. https://doi.org/10.1038/s41539-019-0040-6
 

Students in first-year university courses often focus on mimicking application of taught procedures and fail to gain adequate conceptual understanding. One potential approach to support meaningful learning is Productive Failure (PF). In PF, the conventional instruction process is reversed so that learners attempt to solve challenging problems ahead of receiving explicit instruction. While students often fail to produce satisfactory solutions (hence “Failure”), these attempts help learners encode key features and learn better from subsequent instruction (hence “Productive”). Effectiveness of PF was shown mainly in the context of statistical and intuitive concepts, and lessons that are designed and taught by learning scientists. We describe a quasi-experiment that evaluates the impact of PF in a large-enrollment introductory university-level biology course when designed and implemented by the course instructors. One course-section (295 students) learned two topics using PF; another section (279 students) learned the same topics using an active learning approach, which is the standard in this course. Performance was assessed on the subsequent midterm exam, after all students had ample opportunities for practice and feedback, and after some time has elapsed. PF students scored nearly five percentage-points higher on the relevant topics in the subsequent midterm exam. The effect was especially strong for low-performing students. Improvement on the final exam was only visible for low-performing students. We describe the intervention and its potential to transform large introductory university courses.

@article{b804a177e1944d00874bd2b52f807942,
title = "DIY productive failure: boosting performance in a large undergraduate biology course",
abstract = "Students in first-year university courses often focus on mimicking application of taught procedures and fail to gain adequate conceptual understanding. One potential approach to support meaningful learning is Productive Failure (PF). In PF, the conventional instruction process is reversed so that learners attempt to solve challenging problems ahead of receiving explicit instruction. While students often fail to produce satisfactory solutions (hence “Failure”), these attempts help learners encode key features and learn better from subsequent instruction (hence “Productive”). Effectiveness of PF was shown mainly in the context of statistical and intuitive concepts, and lessons that are designed and taught by learning scientists. We describe a quasi-experiment that evaluates the impact of PF in a large-enrollment introductory university-level biology course when designed and implemented by the course instructors. One course-section (295 students) learned two topics using PF; another section (279 students) learned the same topics using an active learning approach, which is the standard in this course. Performance was assessed on the subsequent midterm exam, after all students had ample opportunities for practice and feedback, and after some time has elapsed. PF students scored nearly five percentage-points higher on the relevant topics in the subsequent midterm exam. The effect was especially strong for low-performing students. Improvement on the final exam was only visible for low-performing students. We describe the intervention and its potential to transform large introductory university courses.",
author = "Chowrira, {Sunita G.} and Smith, {Karen M.} and Dubois, {Patrick J.} and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2019, The Author(s).",
year = "2019",
month = dec,
doi = "10.1038/s41539-019-0040-6",
language = "אנגלית",
volume = "4",
journal = "npj Science of Learning",
issn = "2056-7936",
publisher = "Nature Publishing Group",
number = "1",

}

Weaving together media, technologies and people: Students’ information practices in flipped classrooms

Dodson S, Roll I, Harandi NM, Fels S, Yoon D. Weaving together media, technologies and people: Students’ information practices in flipped classrooms. Information and Learning Science. 2019 Sep 6;120(7-8):519-540. https://doi.org/10.1108/ILS-01-2019-0011
 

Purpose: Students in flipped classrooms are challenged to orchestrate an increasingly heterogeneous collection of learning objects, including audiovisual materials as well as traditional learning objects, such as textbooks and syllabi. This study aims to examine students' information practices interacting with and synthesizing across learning objects, technologies and people in flipped classrooms. Design/methodology/approach: This grounded theory study explores the information practices of 12 undergraduate engineering students as they learned in two flipped classrooms. An artifact walkthrough was used to elicit descriptions of how students conceptualize and work around interoperability problems between the diverse and distributed learning objects by weaving them together into information tapestries. Findings: Students maintained a notebook as an information tapestry, weaving fragmented information snippets from the available learning objects, including, but not limited to, instructional videos and textbooks. Students also connected with peers on Facebook, a back-channel that allowed them to sidestep the academic honesty policy of the course discussion forum, when collaborating on homework assignments. Originality/value: The importance of the interoperability of tools with elements of students' information space and the significance of designing for existing information practices are two outcomes of the grounded theory approach. Design implications for educational technology including the weaving of mixed media and the establishment of spaces for student-to-student interaction are also discussed.

@article{50922644bcb841f29895aafbbaf89df7,
title = "Weaving together media, technologies and people: Students{\textquoteright} information practices in flipped classrooms",
abstract = "Purpose: Students in flipped classrooms are challenged to orchestrate an increasingly heterogeneous collection of learning objects, including audiovisual materials as well as traditional learning objects, such as textbooks and syllabi. This study aims to examine students' information practices interacting with and synthesizing across learning objects, technologies and people in flipped classrooms. Design/methodology/approach: This grounded theory study explores the information practices of 12 undergraduate engineering students as they learned in two flipped classrooms. An artifact walkthrough was used to elicit descriptions of how students conceptualize and work around interoperability problems between the diverse and distributed learning objects by weaving them together into information tapestries. Findings: Students maintained a notebook as an information tapestry, weaving fragmented information snippets from the available learning objects, including, but not limited to, instructional videos and textbooks. Students also connected with peers on Facebook, a back-channel that allowed them to sidestep the academic honesty policy of the course discussion forum, when collaborating on homework assignments. Originality/value: The importance of the interoperability of tools with elements of students' information space and the significance of designing for existing information practices are two outcomes of the grounded theory approach. Design implications for educational technology including the weaving of mixed media and the establishment of spaces for student-to-student interaction are also discussed.",
keywords = "Activity theory, Annotation and note-taking, Flipped classroom, Information practice, Personal information management, Video-based learning",
author = "Samuel Dodson and Ido Roll and Harandi, {Negar M.} and Sidney Fels and Dongwook Yoon",
note = "Publisher Copyright: {\textcopyright} 2019, Emerald Publishing Limited.",
year = "2019",
month = sep,
day = "6",
doi = "10.1108/ILS-01-2019-0011",
language = "אנגלית",
volume = "120",
pages = "519--540",
journal = "Information and Learning Science",
issn = "2398-5348",
publisher = "Emerald Group Publishing Ltd.",
number = "7-8",

}

Instructors desire student activity, literacy, and video quality analytics to improve video-based blended courses

Fong M, Dodson S, Harandi NM, Seo K, Yoon D, Roll I et al. Instructors desire student activity, literacy, and video quality analytics to improve video-based blended courses. In Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Association for Computing Machinery, Inc. 2019. (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019). https://doi.org/10.1145/3330430.3333618
 

While video becomes increasingly prevalent in educational settings, current research has yet to investigate what feedback instructors need regarding their students’ engagement and learning despite video technologies being equipped to provide viewing analytics and collect student feedback. In this paper we investigate instructors’ requirements from video analytics. We used a Grounded Theory Approach and interviewed 16 instructors who teach using video to determine the advantages for using video in their teaching and the different requirements for analytics and feedback in their existing practice. Based on our analysis of the interviews, we found three categories of information that instructors want to inform their teaching. Instructors are looking to see if their students have watched their videos, how much they understood in those videos, and how useful the videos are to the students. These categories provide the foundations and design implications for instructor-centric educational video analytics interfaces.

@inproceedings{39a110cbd81e4a6cb5996ecee204548d,
title = "Instructors desire student activity, literacy, and video quality analytics to improve video-based blended courses",
abstract = "While video becomes increasingly prevalent in educational settings, current research has yet to investigate what feedback instructors need regarding their students{\textquoteright} engagement and learning despite video technologies being equipped to provide viewing analytics and collect student feedback. In this paper we investigate instructors{\textquoteright} requirements from video analytics. We used a Grounded Theory Approach and interviewed 16 instructors who teach using video to determine the advantages for using video in their teaching and the different requirements for analytics and feedback in their existing practice. Based on our analysis of the interviews, we found three categories of information that instructors want to inform their teaching. Instructors are looking to see if their students have watched their videos, how much they understood in those videos, and how useful the videos are to the students. These categories provide the foundations and design implications for instructor-centric educational video analytics interfaces.",
keywords = "Analytics, Blended learning, Learning, Teaching, Video",
author = "Matthew Fong and Samuel Dodson and Harandi, {Negar Mohaghegh} and Kyoungwon Seo and Dongwook Yoon and Ido Roll and Sidney Fels",
note = "Publisher Copyright: {\textcopyright} 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.; 6th ACM Conference on Learning at Scale, L@S 2019 ; Conference date: 24-06-2019 Through 25-06-2019",
year = "2019",
month = jun,
day = "24",
doi = "10.1145/3330430.3333618",
language = "אנגלית",
series = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",
publisher = "Association for Computing Machinery, Inc",
booktitle = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",

}

“Can you believe [1:21]?!”: Content and time-based reference patterns in video comments

Yarmand M, Yoon D, Dodson S, Roll I, Fels SS. “Can you believe [1:21]?!”: Content and time-based reference patterns in video comments. In CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2019. (Conference on Human Factors in Computing Systems - Proceedings). https://doi.org/10.1145/3290605.3300719
 

As videos become increasingly ubiquitous, so is video-based commenting. To contextualize comments, people often reference specific audio/visual content within video. However, the literature falls short of explaining the types of video content people refer to, how they establish references and identify referents, how video characteristics (e.g., genre) impact referencing behaviors, and how references impact social engagement. We present a taxonomy for classifying video references by referent type and temporal specificity. Using our taxonomy, we analyzed 2.5K references with quotations and timestamps collected from public YouTube comments. We found: 1) people reference intervals of video more frequently than time-points, 2) visual entities are referenced more often than sounds, and 3) comments with quotes are more likely to receive replies but not more “likes”. We discuss the need for in-situ dereferencing user interfaces, illustrate design concepts for typed referencing features, and provide a dataset for future studies.

@inproceedings{67208cc3f69c4f9fa083d442fe8198d5,
title = "“Can you believe [1:21]?!”: Content and time-based reference patterns in video comments",
abstract = "As videos become increasingly ubiquitous, so is video-based commenting. To contextualize comments, people often reference specific audio/visual content within video. However, the literature falls short of explaining the types of video content people refer to, how they establish references and identify referents, how video characteristics (e.g., genre) impact referencing behaviors, and how references impact social engagement. We present a taxonomy for classifying video references by referent type and temporal specificity. Using our taxonomy, we analyzed 2.5K references with quotations and timestamps collected from public YouTube comments. We found: 1) people reference intervals of video more frequently than time-points, 2) visual entities are referenced more often than sounds, and 3) comments with quotes are more likely to receive replies but not more “likes”. We discuss the need for in-situ dereferencing user interfaces, illustrate design concepts for typed referencing features, and provide a dataset for future studies.",
keywords = "Comment, Engagement, Reference, Timestamp, Video, YouTube",
author = "Matin Yarmand and Dongwook Yoon and Samuel Dodson and Ido Roll and Fels, {Sidney S.}",
note = "Publisher Copyright: {\textcopyright} 2019 Copyright held by the owner/author(s).; 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 ; Conference date: 04-05-2019 Through 09-05-2019",
year = "2019",
month = may,
day = "2",
doi = "10.1145/3290605.3300719",
language = "אנגלית",
series = "Conference on Human Factors in Computing Systems - Proceedings",
publisher = "Association for Computing Machinery",
booktitle = "CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems",

}

What inquiry with virtual labs can learn from productive failure: A theory-driven study of students’ reflections

Brand C, Massey-Allard J, Perez S, Rummel N, Roll I. What inquiry with virtual labs can learn from productive failure: A theory-driven study of students’ reflections. In Isotani S, Millán E, Ogan A, McLaren B, Hastings P, Luckin R, editors, Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. Springer Verlag. 2019. p. 30-35. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-23207-8_6
 

During inquiry learning with virtual labs students are invited to construct mathematical models that capture key features of the underlying structures. However, students typically fail to construct complete models. In order to identify ways to support learners without restricting them, we look at the literature of Productive Failure and Invention activities (often termed PS-I, Problem Solving before Instruction). PS-I activities are designed to facilitate specific cognitive mechanisms that aid learning. This paper seeks to (1) evaluate in what ways PS-I activities compare to inquiry learning, (2) whether students in inquiry learning report similar processes to PS-I, and (3) whether these are associated with better learning. We begin by synthesizing the two approaches in order to highlight their similarities. Following, we coded self-reported post-activity reflections by 139 students who worked with two virtual labs. Students reported processes that are typical to PS-I and, out of these, prior knowledge activation was associated with constructing more complete models. Based on this, we suggest ways to support students in learning from their inquiry.

@inproceedings{35dfa354c8404dc2a2e4e2a29b3649cc,
title = "What inquiry with virtual labs can learn from productive failure: A theory-driven study of students{\textquoteright} reflections",
abstract = "During inquiry learning with virtual labs students are invited to construct mathematical models that capture key features of the underlying structures. However, students typically fail to construct complete models. In order to identify ways to support learners without restricting them, we look at the literature of Productive Failure and Invention activities (often termed PS-I, Problem Solving before Instruction). PS-I activities are designed to facilitate specific cognitive mechanisms that aid learning. This paper seeks to (1) evaluate in what ways PS-I activities compare to inquiry learning, (2) whether students in inquiry learning report similar processes to PS-I, and (3) whether these are associated with better learning. We begin by synthesizing the two approaches in order to highlight their similarities. Following, we coded self-reported post-activity reflections by 139 students who worked with two virtual labs. Students reported processes that are typical to PS-I and, out of these, prior knowledge activation was associated with constructing more complete models. Based on this, we suggest ways to support students in learning from their inquiry.",
keywords = "Exploratory learning environments, Inquiry learning, Invention activities, Productive failure, Virtual labs",
author = "Charleen Brand and Jonathan Massey-Allard and Sarah Perez and Nikol Rummel and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2019.; 20th International Conference on Artificial Intelligence in Education, AIED 2019 ; Conference date: 25-06-2019 Through 29-06-2019",
year = "2019",
doi = "10.1007/978-3-030-23207-8_6",
language = "אנגלית",
isbn = "9783030232061",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "30--35",
editor = "Seiji Isotani and Eva Mill{\'a}n and Amy Ogan and Bruce McLaren and Peter Hastings and Rose Luckin",
booktitle = "Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings",

}

Supporting productive exploration in invention activities: Are simulations too challenging?

Massey-Allard J, Roll I, Ives J. Supporting productive exploration in invention activities: Are simulations too challenging? In Cao Y, Wolf S, Bennett M, editors, Physics Education Research Conference, PERC 2019. American Association of Physics Teachers. 2019. p. 372-377. (Physics Education Research Conference Proceedings). https://doi.org/10.1119/perc.2019.pr.Massey-Allard
 

Studies show that invention activities, where students invent a general rule from provided resources before receiving direct instruction on the target topic, are particularly beneficial for learning outcomes. For most common implementations of invention activities, students are provided with instructor-designed contrasting cases with which to invent their rule. Alternatively, students could use an interactive simulation where they then have the agency to explore and collect observations on their own. While this provides a promising opportunity for developing more robust inquiry skills, it also introduces substantial challenges for the students that, in addition to learning about the domain, need to learn about expert ways of doing science. In this work, we compare different support structures that seek to mitigate these issues. Specifically, we incorporate a collaborative support structure and further provide students with either a short list of general rules to disprove or a list of important features that students are prompted to incorporate in their rule. We show that these support structures are not sufficient to make the exploration of students in our simulation-based invention activities as productive as with using contrasting cases.

@inproceedings{b1f9a5bd9ae741dfb7a3e7513702dcd1,
title = "Supporting productive exploration in invention activities: Are simulations too challenging?",
abstract = "Studies show that invention activities, where students invent a general rule from provided resources before receiving direct instruction on the target topic, are particularly beneficial for learning outcomes. For most common implementations of invention activities, students are provided with instructor-designed contrasting cases with which to invent their rule. Alternatively, students could use an interactive simulation where they then have the agency to explore and collect observations on their own. While this provides a promising opportunity for developing more robust inquiry skills, it also introduces substantial challenges for the students that, in addition to learning about the domain, need to learn about expert ways of doing science. In this work, we compare different support structures that seek to mitigate these issues. Specifically, we incorporate a collaborative support structure and further provide students with either a short list of general rules to disprove or a list of important features that students are prompted to incorporate in their rule. We show that these support structures are not sufficient to make the exploration of students in our simulation-based invention activities as productive as with using contrasting cases.",
author = "Jonathan Massey-Allard and Ido Roll and Joss Ives",
note = "Publisher Copyright: {\textcopyright} 2019, American Association of Physics Teachers. All rights reserved.; Physics Education Research Conference, PERC 2019 ; Conference date: 24-07-2019 Through 25-07-2019",
year = "2019",
doi = "10.1119/perc.2019.pr.Massey-Allard",
language = "אנגלית",
isbn = "9781931024365",
series = "Physics Education Research Conference Proceedings",
publisher = "American Association of Physics Teachers",
pages = "372--377",
editor = "Ying Cao and Steven Wolf and Michael Bennett",
booktitle = "Physics Education Research Conference, PERC 2019",
address = "ארצות הברית",

}

2018

Learning at Scale

Roll I, Russell DM, Gašević D. Learning at Scale. International Journal of Artificial Intelligence in Education. 2018 Sep 1;28(4):471-477. https://doi.org/10.1007/s40593-018-0170-7
 

Learning at Scale is a fast growing field that affects formal, informal, and workplace education. Highly interdisciplinary, it builds on solid foundations in the learning sciences, computer science, education, and the social sciences. We define learning at scale as the study of the technologies, pedagogies, analyses, and theories of learning and teaching that take place with a large number of learners and a high ratio of learners to facilitators. The scale of these environments often changes the very nature of the interaction and learning experiences. We identify three types of technologies that support scale in education: dedicated content-agnostic platforms, such as MOOCs; dedicated tools, such as Intelligent Tutoring Systems; and repurposed platforms, such as social networks. We further identify five areas that scale affects: learners, research and data, adaptation, space and time, and pedagogy. Introducing the papers in this special issue on the topic, we discuss the characteristics, affordances, and promise of learning at scale.

@article{bb2b7e60b44e4a7ab67d1e7eb96a3814,
title = "Learning at Scale",
abstract = "Learning at Scale is a fast growing field that affects formal, informal, and workplace education. Highly interdisciplinary, it builds on solid foundations in the learning sciences, computer science, education, and the social sciences. We define learning at scale as the study of the technologies, pedagogies, analyses, and theories of learning and teaching that take place with a large number of learners and a high ratio of learners to facilitators. The scale of these environments often changes the very nature of the interaction and learning experiences. We identify three types of technologies that support scale in education: dedicated content-agnostic platforms, such as MOOCs; dedicated tools, such as Intelligent Tutoring Systems; and repurposed platforms, such as social networks. We further identify five areas that scale affects: learners, research and data, adaptation, space and time, and pedagogy. Introducing the papers in this special issue on the topic, we discuss the characteristics, affordances, and promise of learning at scale.",
keywords = "Intelligent tutoring systems, Learning analytics, Learning at scale, MOOCs",
author = "Ido Roll and Russell, {Daniel M.} and Dragan Ga{\v s}evi{\'c}",
note = "Publisher Copyright: {\textcopyright} 2018, International Artificial Intelligence in Education Society.",
year = "2018",
month = sep,
day = "1",
doi = "10.1007/s40593-018-0170-7",
language = "אנגלית",
volume = "28",
pages = "471--477",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "4",

}

An active viewing framework for video-based learning

Dodson S, Roll I, Fong M, Yoon D, Harandi NM, Fels S. An active viewing framework for video-based learning. In Proceedings of the 5th Annual ACM Conference on Learning at Scale, L at S 2018. Association for Computing Machinery, Inc. 2018. 24. (Proceedings of the 5th Annual ACM Conference on Learning at Scale, L at S 2018). https://doi.org/10.1145/3231644.3231682
 

Video-based learning is most effective when students are engaged with video content; however, the literature has yet to identify students' viewing behaviors and ground them in theory. This paper addresses this need by introducing a framework of active viewing, which is situated in an established model of active learning to describe students' behaviors while learning from video. We conducted a field study with 460 undergraduates in an Applied Science course using a video player designed for active viewing to evaluate how students engage in passive and active video-based learning. The concept of active viewing, and the role of interactive, constructive, active, and passive behaviors in videobased learning, can be implemented in the design and evaluation of video players.

@inproceedings{41d9d29385e846ddad5b6639c69ed214,
title = "An active viewing framework for video-based learning",
abstract = "Video-based learning is most effective when students are engaged with video content; however, the literature has yet to identify students' viewing behaviors and ground them in theory. This paper addresses this need by introducing a framework of active viewing, which is situated in an established model of active learning to describe students' behaviors while learning from video. We conducted a field study with 460 undergraduates in an Applied Science course using a video player designed for active viewing to evaluate how students engage in passive and active video-based learning. The concept of active viewing, and the role of interactive, constructive, active, and passive behaviors in videobased learning, can be implemented in the design and evaluation of video players.",
keywords = "Active Viewing, Annotation, Video-Based Learning",
author = "Samuel Dodson and Ido Roll and Matthew Fong and Dongwook Yoon and Harandi, {Negar M.} and Sidney Fels",
note = "Publisher Copyright: {\textcopyright} 2017 Association for Computing Machinery. All rights reserved.; 5th Annual ACM Conference on Learning at Scale, L at S 2018 ; Conference date: 26-06-2018 Through 28-06-2018",
year = "2018",
month = jun,
day = "26",
doi = "10.1145/3231644.3231682",
language = "אנגלית",
series = "Proceedings of the 5th Annual ACM Conference on Learning at Scale, L at S 2018",
publisher = "Association for Computing Machinery, Inc",
booktitle = "Proceedings of the 5th Annual ACM Conference on Learning at Scale, L at S 2018",

}

ViDeX: A Platform for Personalizing Educational Videos

Fong M, Dodson S, Zhang X, Roll I, Fels S. ViDeX: A Platform for Personalizing Educational Videos. In JCDL 2018 - Proceedings of the 18th ACM/IEEE Joint Conference on Digital Libraries. Institute of Electrical and Electronics Engineers Inc. 2018. p. 331-332. (Proceedings of the ACM/IEEE Joint Conference on Digital Libraries). https://doi.org/10.1145/3197026.3203865
 

As video-based learning is increasingly used in all sectors of education, there is a need for video players that support active viewing practices. We introduce a video player that allows students to mark up video with highlights, tags, and notes in order to personalize their video-based learning experience.

@inproceedings{0849878e339f4c4180067872e7f6d714,
title = "ViDeX: A Platform for Personalizing Educational Videos",
abstract = "As video-based learning is increasingly used in all sectors of education, there is a need for video players that support active viewing practices. We introduce a video player that allows students to mark up video with highlights, tags, and notes in order to personalize their video-based learning experience.",
keywords = "active viewing, annotation, personalization, video-based learning",
author = "Matthew Fong and Samuel Dodson and Xueqin Zhang and Ido Roll and Sidney Fels",
note = "Publisher Copyright: {\textcopyright} 2018 Authors.; 18th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2018 ; Conference date: 03-06-2018 Through 07-06-2018",
year = "2018",
month = may,
day = "23",
doi = "10.1145/3197026.3203865",
language = "אנגלית",
series = "Proceedings of the ACM/IEEE Joint Conference on Digital Libraries",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "331--332",
booktitle = "JCDL 2018 - Proceedings of the 18th ACM/IEEE Joint Conference on Digital Libraries",

}

Active viewing: A study of video highlighting in the classroom

Dodson S, Yoon D, Roll I, Harandi NM, Fong M, Fels S. Active viewing: A study of video highlighting in the classroom. In CHIIR 2018 - Proceedings of the 2018 Conference on Human Information Interaction and Retrieval. Association for Computing Machinery, Inc. 2018. p. 237-240. (CHIIR 2018 - Proceedings of the 2018 Conference on Human Information Interaction and Retrieval). https://doi.org/10.1145/3176349.3176889
 

Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners’ active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.

@inproceedings{1bb3e6ae759f4735982b895bb5fac14d,
title = "Active viewing: A study of video highlighting in the classroom",
abstract = "Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners{\textquoteright} active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.",
keywords = "Active viewing, Highlighting, Learning, Video search",
author = "Samuel Dodson and Dongwook Yoon and Ido Roll and Harandi, {Negar M.} and Matthew Fong and Sidney Fels",
note = "Publisher Copyright: {\textcopyright} 2018 Copyright held by the owner/author(s).; 3rd ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR 2018 ; Conference date: 11-03-2018 Through 15-03-2018",
year = "2018",
month = feb,
day = "1",
doi = "10.1145/3176349.3176889",
language = "אנגלית",
series = "CHIIR 2018 - Proceedings of the 2018 Conference on Human Information Interaction and Retrieval",
publisher = "Association for Computing Machinery, Inc",
pages = "237--240",
booktitle = "CHIIR 2018 - Proceedings of the 2018 Conference on Human Information Interaction and Retrieval",

}

Understanding the impact of guiding inquiry: the relationship between directive support, student attributes, and transfer of knowledge, attitudes, and behaviours in inquiry learning

Roll I, Butler D, Yee N, Welsh A, Perez S, Briseno A et al. Understanding the impact of guiding inquiry: the relationship between directive support, student attributes, and transfer of knowledge, attitudes, and behaviours in inquiry learning. Instructional Science. 2018 Feb 1;46(1):77-104. https://doi.org/10.1007/s11251-017-9437-x
 

Guiding inquiry learning has been shown to increase knowledge gains. Yet, little is known about the effect of guidance on attitudes and behaviours, its interaction with student attributes, and transfer of impact once guidance is removed. We address these gaps in the context of an interactive Physics simulation on electric circuits (https://phet.colorado.edu/en/simulation/circuit-construction-kit-dc). 49 students in the Non-Directive condition received a set of goals to focus their inquiry, in addition to implicit support built into the simulation. 48 students in the Directive condition received, in addition to these, also detailed directions and prompts. Log-file analysis found that directive support led to more formal testing and less exploration. Clustering identified two groups of learners: one with higher incoming knowledge (Higher Knowledge), the other with higher incoming perceptions of competence and control (Higher PoCC). Working with the simulation improved knowledge and attitudes across cluster groups, so that prior differences all but disappeared. With regard to guidance, adding directive support improved knowledge gains for the Higher Knowledge group, yet suppressed their attitudinal growth. The same support had no effect on knowledge gains for the Higher PoCC group, yet it boosted their attitudinal growth. A transfer activity, where directive support was no longer available, found that impact on attitudes carried forward, yet impacts on behaviours and knowledge were short-lived. Overall, the study highlights the complex interaction between guidance and student attributes. For some, supporting short-term knowledge gains may inadvertenly lead to longer term negative impact on attitudes towards inquiry.

@article{e95e67e03d8b493a92f9f522ed9ca987,
title = "Understanding the impact of guiding inquiry: the relationship between directive support, student attributes, and transfer of knowledge, attitudes, and behaviours in inquiry learning",
abstract = "Guiding inquiry learning has been shown to increase knowledge gains. Yet, little is known about the effect of guidance on attitudes and behaviours, its interaction with student attributes, and transfer of impact once guidance is removed. We address these gaps in the context of an interactive Physics simulation on electric circuits (https://phet.colorado.edu/en/simulation/circuit-construction-kit-dc). 49 students in the Non-Directive condition received a set of goals to focus their inquiry, in addition to implicit support built into the simulation. 48 students in the Directive condition received, in addition to these, also detailed directions and prompts. Log-file analysis found that directive support led to more formal testing and less exploration. Clustering identified two groups of learners: one with higher incoming knowledge (Higher Knowledge), the other with higher incoming perceptions of competence and control (Higher PoCC). Working with the simulation improved knowledge and attitudes across cluster groups, so that prior differences all but disappeared. With regard to guidance, adding directive support improved knowledge gains for the Higher Knowledge group, yet suppressed their attitudinal growth. The same support had no effect on knowledge gains for the Higher PoCC group, yet it boosted their attitudinal growth. A transfer activity, where directive support was no longer available, found that impact on attitudes carried forward, yet impacts on behaviours and knowledge were short-lived. Overall, the study highlights the complex interaction between guidance and student attributes. For some, supporting short-term knowledge gains may inadvertenly lead to longer term negative impact on attitudes towards inquiry.",
keywords = "Assistance dilemma, Exploratory learning environments, Inquiry learning, discovery-based learning, Interactive simulations, Scaffolding, Transfer",
author = "Ido Roll and Deborah Butler and Nikki Yee and Ashley Welsh and Sarah Perez and Adriana Briseno and Katherine Perkins and Doug Bonn",
note = "Publisher Copyright: {\textcopyright} 2017, Springer Science+Business Media B.V., part of Springer Nature.",
year = "2018",
month = feb,
day = "1",
doi = "10.1007/s11251-017-9437-x",
language = "אנגלית",
volume = "46",
pages = "77--104",
journal = "Instructional Science",
issn = "0020-4277",
publisher = "Springer Netherlands",
number = "1",

}

Control of variables strategy across phases of inquiry in virtual labs

Perez S, Massey-Allard J, Ives J, Butler D, Bonn D, Bale J et al. Control of variables strategy across phases of inquiry in virtual labs. In Luckin R, Porayska-Pomsta K, du Boulay B, Mavrikis M, Penstein Rosé C, McLaren B, Martinez-Maldonado R, Hoppe HU, editors, Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings. Springer Verlag. 2018. p. 271-275. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-93846-2_50
 

Control of Variables Strategy (CVS) is the process of isolating the effect of single variables when conducting scientific inquiry. We assess how CVS can help student achieve different levels of understanding when implemented in different parts of the inquiry process. 148 students worked with minimally-guided inquiry activities using virtual labs on two different physics topics. The virtual labs allowed for exploration, data collection, and graphical analysis. Using student log data, we identified how CVS manifests itself through these phases of students’ inquiry process. We found that students using CVS during data collection and plotting was associated with students achieving more qualitative and quantitative models, respectively. This did not hold, however, for more complicated mathematical relationships, emphasizing the importance of mathematical and graphical interpretation skills when doing CVS.

@inproceedings{a30fb9d259724eb5ae7d30f267f4d34e,
title = "Control of variables strategy across phases of inquiry in virtual labs",
abstract = "Control of Variables Strategy (CVS) is the process of isolating the effect of single variables when conducting scientific inquiry. We assess how CVS can help student achieve different levels of understanding when implemented in different parts of the inquiry process. 148 students worked with minimally-guided inquiry activities using virtual labs on two different physics topics. The virtual labs allowed for exploration, data collection, and graphical analysis. Using student log data, we identified how CVS manifests itself through these phases of students{\textquoteright} inquiry process. We found that students using CVS during data collection and plotting was associated with students achieving more qualitative and quantitative models, respectively. This did not hold, however, for more complicated mathematical relationships, emphasizing the importance of mathematical and graphical interpretation skills when doing CVS.",
keywords = "Control variable strategies, Inquiry learning, Virtual lab",
author = "Sarah Perez and Jonathan Massey-Allard and Joss Ives and Deborah Butler and Doug Bonn and Jeff Bale and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG, part of Springer Nature 2018.; 19th International Conference on Artificial Intelligence in Education, AIED 2018 ; Conference date: 27-06-2018 Through 30-06-2018",
year = "2018",
doi = "10.1007/978-3-319-93846-2_50",
language = "אנגלית",
isbn = "9783319938455",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "271--275",
editor = "Rose Luckin and Kaska Porayska-Pomsta and {du Boulay}, Benedict and Manolis Mavrikis and {Penstein Ros{\'e}}, Carolyn and Bruce McLaren and Roberto Martinez-Maldonado and Hoppe, {H. Ulrich}",
booktitle = "Artificial Intelligence in Education - 19th International Conference, AIED 2018, Proceedings",

}

Student Video-Usage in Introductory Engineering Courses

Harandi NM, Agharebparast F, Linares L, Dodson S, Roll I, Fong M et al. Student Video-Usage in Introductory Engineering Courses. Proceedings of the Canadian Engineering Education Association (CEEA). 2018. https://doi.org/10.24908/pceea.v0i0.13025
 
As videos are gaining popularity in flipped and blended Engineering classrooms, there is an increasing need to track and understand students’ use of the videos, in order to identify evidence-based practices matched to the emerging trends in video and video annotation tools. We explore students’ surveyresponses, follow-up interviews, and log data from their interaction with common video platforms as well as, ViDeX, a new experimental video annotation tool, to evaluate how, when and why students watch, rewatch, and annotate videos in two large introductory Engineering courses, with flipped, and blended formats. Our findings show that students watch thevideos with the instructors’ intended use in mind, and plan their review process accordingly. In the flipped classroom, most students summarized the short preclass screencasts in their personal notes to minimize the need to re-watch the videos before the exam. In contrast, students in the blended  classroom reexamined the long tutorial videos mostly to redo the problems before the midterm and final exams. Bookmarking seemed to be useful for locating those problems of interest. Since the problems required drawings and computations, paper annotation was more beneficial than a dedicated video annotation platform.
@article{494dc49d93de4827886346ab069daeb2,
title = "Student Video-Usage in Introductory Engineering Courses",
abstract = "As videos are gaining popularity in flipped and blended Engineering classrooms, there is an increasing need to track and understand students{\textquoteright} use of the videos, in order to identify evidence-based practices matched to the emerging trends in video and video annotation tools. We explore students{\textquoteright} surveyresponses, follow-up interviews, and log data from their interaction with common video platforms as well as, ViDeX, a new experimental video annotation tool, to evaluate how, when and why students watch, rewatch, and annotate videos in two large introductory Engineering courses, with flipped, and blended formats. Our findings show that students watch thevideos with the instructors{\textquoteright} intended use in mind, and plan their review process accordingly. In the flipped classroom, most students summarized the short preclass screencasts in their personal notes to minimize the need to re-watch the videos before the exam. In contrast, students in the blended  classroom reexamined the long tutorial videos mostly to redo the problems before the midterm and final exams. Bookmarking seemed to be useful for locating those problems of interest. Since the problems required drawings and computations, paper annotation was more beneficial than a dedicated video annotation platform.",
author = "Harandi, {Negar M.} and Farshid Agharebparast and Luis Linares and Samuel Dodson and Ido Roll and Matthew Fong and Dongwook Yoon and Sidney Fels",
year = "2018",
doi = "10.24908/pceea.v0i0.13025",
language = "American English",
journal = "Proceedings of the Canadian Engineering Education Association (CEEA)",

}

2017

Towards a Theory of When and How Problem Solving Followed by Instruction Supports Learning

Loibl K, Roll I, Rummel N. Towards a Theory of When and How Problem Solving Followed by Instruction Supports Learning. Educational Psychology Review. 2017 Dec 1;29(4):693-715. https://doi.org/10.1007/s10648-016-9379-x
 

Recently, there has been a growing interest in learning approaches that combine two phases: an initial problem-solving phase followed by an instruction phase (PS-I). Two often cited examples of instructional approaches following the PS-I scheme include Productive Failure and Invention. Despite the growing interest in PS-I approaches, to the best of our knowledge, there has not yet been a comprehensive attempt to summarize the features that define PS-I and to explain the patterns of results. Therefore, the first goal of this paper is to map the landscape of different PS-I implementations, to identify commonalities and differences in designs, and to associate the identified design features with patterns in the learning outcomes. The review shows that PS-I fosters learning only if specific design features (namely contrasting cases or building instruction on student solutions) are implemented. The second goal is to identify a set of interconnected cognitive mechanisms that may account for these outcomes. Empirical evidence from PS-I literature is associated with these mechanisms and supports an initial theory of PS-I. Finally, positive and negative effects of PS-I are explained using the suggested mechanisms.

@article{4777892affef4b8888bcb4d16926f67d,
title = "Towards a Theory of When and How Problem Solving Followed by Instruction Supports Learning",
abstract = "Recently, there has been a growing interest in learning approaches that combine two phases: an initial problem-solving phase followed by an instruction phase (PS-I). Two often cited examples of instructional approaches following the PS-I scheme include Productive Failure and Invention. Despite the growing interest in PS-I approaches, to the best of our knowledge, there has not yet been a comprehensive attempt to summarize the features that define PS-I and to explain the patterns of results. Therefore, the first goal of this paper is to map the landscape of different PS-I implementations, to identify commonalities and differences in designs, and to associate the identified design features with patterns in the learning outcomes. The review shows that PS-I fosters learning only if specific design features (namely contrasting cases or building instruction on student solutions) are implemented. The second goal is to identify a set of interconnected cognitive mechanisms that may account for these outcomes. Empirical evidence from PS-I literature is associated with these mechanisms and supports an initial theory of PS-I. Finally, positive and negative effects of PS-I are explained using the suggested mechanisms.",
keywords = "Compare and contrast, Contrasting cases, Invention, Learning mechanisms, Problem solving, Productive Failure, Student solutions",
author = "Katharina Loibl and Ido Roll and Nikol Rummel",
note = "Publisher Copyright: {\textcopyright} 2016, Springer Science+Business Media New York.",
year = "2017",
month = dec,
day = "1",
doi = "10.1007/s10648-016-9379-x",
language = "אנגלית",
volume = "29",
pages = "693--715",
journal = "Educational Psychology Review",
issn = "1040-726X",
publisher = "Springer New York",
number = "4",

}

Applying a Framework for Student Modeling in Exploratory Learning Environments: Comparing Data Representation Granularity to Handle Environment Complexity

Fratamico L, Conati C, Kardan S, Roll I. Applying a Framework for Student Modeling in Exploratory Learning Environments: Comparing Data Representation Granularity to Handle Environment Complexity. International Journal of Artificial Intelligence in Education. 2017 Jun 1;27(2):320-352. https://doi.org/10.1007/s40593-016-0131-y
 

Interactive simulations can facilitate inquiry learning. However, similarly to other Exploratory Learning Environments, students may not always learn effectively in these unstructured environments. Thus, providing adaptive support has great potential to help improve student learning with these rich activities. Providing adaptive support requires a student model that can both evaluate learning as well inform relevant feedback. Building such a model for interactive simulations is especially challenging because the exploratory nature of the interaction makes it hard to know a priori which behaviors are conducive to learning. To address this problem, in this paper we leverage the student modeling framework proposed in (Kardan and Conati, 2011) to specifically address the challenge of modeling students in interactive simulations. The framework has already been successfully applied to build a student model and to give adaptive interventions for an interactive simulation for constraint satisfaction. We seek to investigate the generality of the framework by building student models for a more complex simulation on electric circuits called Circuit Construction Kit (CCK). We evaluate alternative representations of logged interaction data with CCK, capturing different amounts of granularity and feature engineering. We then apply the student modeling framework proposed in (Kardan and Conati, 2011) to group students based on their interaction behaviors, map these behaviors into learning outcomes and leverage the resulting clusters to classify new learners. Data collected from 100 college students working with the CCK simulation indicates that the proposed framework is able to successfully classify students in groups of high and low learners and identify patterns of productive behaviors that are common across representations that can inform real-time feedback. In addition to presenting these results, we discuss trade-offs between levels of granularity and feature engineering in the tested interaction representations in terms of their ability to evaluate learning, classify students, and inform feedback.

@article{ae45b9e20ad147d494e3d3bcfe42170e,
title = "Applying a Framework for Student Modeling in Exploratory Learning Environments: Comparing Data Representation Granularity to Handle Environment Complexity",
abstract = "Interactive simulations can facilitate inquiry learning. However, similarly to other Exploratory Learning Environments, students may not always learn effectively in these unstructured environments. Thus, providing adaptive support has great potential to help improve student learning with these rich activities. Providing adaptive support requires a student model that can both evaluate learning as well inform relevant feedback. Building such a model for interactive simulations is especially challenging because the exploratory nature of the interaction makes it hard to know a priori which behaviors are conducive to learning. To address this problem, in this paper we leverage the student modeling framework proposed in (Kardan and Conati, 2011) to specifically address the challenge of modeling students in interactive simulations. The framework has already been successfully applied to build a student model and to give adaptive interventions for an interactive simulation for constraint satisfaction. We seek to investigate the generality of the framework by building student models for a more complex simulation on electric circuits called Circuit Construction Kit (CCK). We evaluate alternative representations of logged interaction data with CCK, capturing different amounts of granularity and feature engineering. We then apply the student modeling framework proposed in (Kardan and Conati, 2011) to group students based on their interaction behaviors, map these behaviors into learning outcomes and leverage the resulting clusters to classify new learners. Data collected from 100 college students working with the CCK simulation indicates that the proposed framework is able to successfully classify students in groups of high and low learners and identify patterns of productive behaviors that are common across representations that can inform real-time feedback. In addition to presenting these results, we discuss trade-offs between levels of granularity and feature engineering in the tested interaction representations in terms of their ability to evaluate learning, classify students, and inform feedback.",
keywords = "Clustering, Educational data mining, Exploratory learning environments, Interactive simulations, User modeling",
author = "Lauren Fratamico and Cristina Conati and Samad Kardan and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2017, International Artificial Intelligence in Education Society.",
year = "2017",
month = jun,
day = "1",
doi = "10.1007/s40593-016-0131-y",
language = "אנגלית",
volume = "27",
pages = "320--352",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "2",

}

A visual approach towards knowledge engineering and understanding how students learn in complex environments

Fratamico L, Perez S, Roll I. A visual approach towards knowledge engineering and understanding how students learn in complex environments. In L@S 2017 - Proceedings of the 4th (2017) ACM Conference on Learning at Scale. Association for Computing Machinery, Inc. 2017. p. 13-22. (L@S 2017 - Proceedings of the 4th (2017) ACM Conference on Learning at Scale). https://doi.org/10.1145/3051457.3051468
 

Exploratory learning environments, such as virtual labs, support divergent learning pathways. However, due to their complexity, building computational models of learning is challenging as it is difficult to identify features that (i) are informative with respect to common learning strategies, (ii) abstract similar actions beyond surface differences, and (iii) differentiate groups of learners. In this paper, we present a visualization tool that addresses these challenges by facilitating a novel analytic approach to aid in the knowledge engineering process, focusing on five main capabilities: data-driven hypotheses raising, visualizing behavior over time, easily grouping related actions, contrasting learners' behaviors on these actions, and comparing the behaviors of groups of learners. We apply this analytic approach to better understand how students work with a popular interactive physics virtual lab. By splitting learners by learning gains, we found that productive learners performed more active testing and adapted more quickly to the task at hand by focusing on more relevant testing instruments. Implications for online virtual labs and a broader class of complex learning environments are discussed throughout.

@inproceedings{383e81174eae4e43a9b0a9be11fec0d1,
title = "A visual approach towards knowledge engineering and understanding how students learn in complex environments",
abstract = "Exploratory learning environments, such as virtual labs, support divergent learning pathways. However, due to their complexity, building computational models of learning is challenging as it is difficult to identify features that (i) are informative with respect to common learning strategies, (ii) abstract similar actions beyond surface differences, and (iii) differentiate groups of learners. In this paper, we present a visualization tool that addresses these challenges by facilitating a novel analytic approach to aid in the knowledge engineering process, focusing on five main capabilities: data-driven hypotheses raising, visualizing behavior over time, easily grouping related actions, contrasting learners' behaviors on these actions, and comparing the behaviors of groups of learners. We apply this analytic approach to better understand how students work with a popular interactive physics virtual lab. By splitting learners by learning gains, we found that productive learners performed more active testing and adapted more quickly to the task at hand by focusing on more relevant testing instruments. Implications for online virtual labs and a broader class of complex learning environments are discussed throughout.",
keywords = "H.5.3. Information interfaces and presentation (e.g. HCI): Group and organization interfaces, K.3.1. computers and education: Computer uses in education",
author = "Lauren Fratamico and Sarah Perez and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2017 ACM.; 4th Annual ACM Conference on Learning at Scale, L@S 2017 ; Conference date: 20-04-2017 Through 21-04-2017",
year = "2017",
month = apr,
day = "12",
doi = "10.1145/3051457.3051468",
language = "אנגלית",
series = "L@S 2017 - Proceedings of the 4th (2017) ACM Conference on Learning at Scale",
publisher = "Association for Computing Machinery, Inc",
pages = "13--22",
booktitle = "L@S 2017 - Proceedings of the 4th (2017) ACM Conference on Learning at Scale",

}

Compair: A new online tool using adaptive comparative judgement to support learning with peer feedback

Potter T, Englund L, Charbonneau J, MacLean MT, Newell J, Roll I. Compair: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching and Learning Inquiry. 2017;5(2):89-113. https://doi.org/10.20343/teachlearninqu.5.2.8
 

Peer feedback is a useful strategy in teaching and learning, but its effectiveness particularly in introductory courses can be limited by the relative newness of students to both the body of knowledge upon which they are being asked to provide feedback and the skill set involved in providing good feedback. This paper applies a novel approach to facilitating novice feedback: making use of students' inherent ability to compare. The ComPAIR application discussed in this article scaffolds peer feedback through comparisons, asking students to choose the “better” of two answers in a series of pairings offered in an engaging online context. In contrast to other peer-feedback approaches that seek to train novices to be able to provide expert feedback (such as calibrated peer review) or to crowdsource grading, ComPAIR focuses upon the benefits to be gained from the critical process of comparison and ranking. The tool design is based on the longstanding psychological principle of comparative judgement, by which novices who may not yet have the compass to assess others' work confidently can still rank content as “better” with accuracy. Data from 168 students in pilot studies in English, Physics and Math courses at the University of British Columbia are reviewed. Though the use of ComPAIR required little classroom time, students perceived this approach to increase their facility with course content, their ability assess their own work, and their capacity to provide feedback on the work of others in a collaborative learning environment.

@article{95c0172f29c54aee8bc2092c31d4595e,
title = "Compair: A new online tool using adaptive comparative judgement to support learning with peer feedback",
abstract = "Peer feedback is a useful strategy in teaching and learning, but its effectiveness particularly in introductory courses can be limited by the relative newness of students to both the body of knowledge upon which they are being asked to provide feedback and the skill set involved in providing good feedback. This paper applies a novel approach to facilitating novice feedback: making use of students' inherent ability to compare. The ComPAIR application discussed in this article scaffolds peer feedback through comparisons, asking students to choose the “better” of two answers in a series of pairings offered in an engaging online context. In contrast to other peer-feedback approaches that seek to train novices to be able to provide expert feedback (such as calibrated peer review) or to crowdsource grading, ComPAIR focuses upon the benefits to be gained from the critical process of comparison and ranking. The tool design is based on the longstanding psychological principle of comparative judgement, by which novices who may not yet have the compass to assess others' work confidently can still rank content as “better” with accuracy. Data from 168 students in pilot studies in English, Physics and Math courses at the University of British Columbia are reviewed. Though the use of ComPAIR required little classroom time, students perceived this approach to increase their facility with course content, their ability assess their own work, and their capacity to provide feedback on the work of others in a collaborative learning environment.",
keywords = "Adaptive comparative judgement, Answer comparison, Collaborative learning, Online teaching tools, Peer feedback",
author = "Tiffany Potter and Letitia Englund and James Charbonneau and MacLean, {Mark Thompson} and Jonathan Newell and Ido Roll",
note = "Publisher Copyright: Copyright {\textcopyright} 2017 The International Society for the Scholarship of Teaching and Learning",
year = "2017",
doi = "10.20343/teachlearninqu.5.2.8",
language = "אנגלית",
volume = "5",
pages = "89--113",
journal = "Teaching and Learning Inquiry",
issn = "2167-4787",
publisher = "University of Calgary Press",
number = "2",

}

Identifying productive inquiry in virtual labs using sequence mining

Perez S, Massey-Allard J, Butler D, Ives J, Bonn D, Yee N et al. Identifying productive inquiry in virtual labs using sequence mining. In Andre E, Hu X, Rodrigo MMT, du Boulay B, Baker R, editors, Artificial Intelligence in Education - 18th International Conference, AIED 2017, Proceedings. Springer Verlag. 2017. p. 287-298. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-61425-0_24
 

Virtual labs are exploratory learning environments in which students learn by conducting inquiry to uncover the underlying scientific model. Although students often fail to learn efficiently in these environments, providing effective support is challenging since it is unclear what productive engagement looks like. This paper focuses on the mining and identification of student inquiry strategies during an unstructured activity with the DC Circuit Construction Kit (https://phet.colorado.edu/). We use an information theoretic sequence mining method to identify productive and unproductive strategies of a hundred students. Low domain knowledge students who successfully learned during the activity paused more after testing their circuits, particularly on simply structured circuits that target the activity’s learning goals, and mainly earlier in the activity. Moreover, our results show that a strategic use of pauses so that they become opportunities for reflection and planning is highly associated with productive learning. Implication to theory, support, and assessment are discussed.

@inproceedings{1b5fb6f0da4d43a198ac9316ef6b20c9,
title = "Identifying productive inquiry in virtual labs using sequence mining",
abstract = "Virtual labs are exploratory learning environments in which students learn by conducting inquiry to uncover the underlying scientific model. Although students often fail to learn efficiently in these environments, providing effective support is challenging since it is unclear what productive engagement looks like. This paper focuses on the mining and identification of student inquiry strategies during an unstructured activity with the DC Circuit Construction Kit (https://phet.colorado.edu/). We use an information theoretic sequence mining method to identify productive and unproductive strategies of a hundred students. Low domain knowledge students who successfully learned during the activity paused more after testing their circuits, particularly on simply structured circuits that target the activity{\textquoteright}s learning goals, and mainly earlier in the activity. Moreover, our results show that a strategic use of pauses so that they become opportunities for reflection and planning is highly associated with productive learning. Implication to theory, support, and assessment are discussed.",
keywords = "Exploratory learning environments, Inquiry learning, Self-regulated learning, Sequence mining, Virtual lab",
author = "Sarah Perez and Jonathan Massey-Allard and Deborah Butler and Joss Ives and Doug Bonn and Nikki Yee and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017.; 18th International Conference on Artificial Intelligence in Education, AIED 2017 ; Conference date: 28-06-2017 Through 01-07-2017",
year = "2017",
doi = "10.1007/978-3-319-61425-0_24",
language = "אנגלית",
isbn = "9783319614243",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "287--298",
editor = "Elisabeth Andre and Xiangen Hu and Rodrigo, {Ma. Mercedes T.} and {du Boulay}, Benedict and Ryan Baker",
booktitle = "Artificial Intelligence in Education - 18th International Conference, AIED 2017, Proceedings",

}

2016

Evolution and Revolution in Artificial Intelligence in Education

Roll I, Wylie R. Evolution and Revolution in Artificial Intelligence in Education. International Journal of Artificial Intelligence in Education. 2016 Jun 1;26(2):582-599. https://doi.org/10.1007/s40593-016-0110-3
 

The field of Artificial Intelligence in Education (AIED) has undergone significant developments over the last twenty-five years. As we reflect on our past and shape our future, we ask two main questions: What are our major strengths? And, what new opportunities lay on the horizon? We analyse 47 papers from three years in the history of the Journal of AIED (1994, 2004, and 2014) to identify the foci and typical scenarios that occupy the field of AIED. We use those results to suggest two parallel strands of research that need to take place in order to impact education in the next 25 years: One is an evolutionary process, focusing on current classroom practices, collaborating with teachers, and diversifying technologies and domains. The other is a revolutionary process where we argue for embedding our technologies within students' everyday lives, supporting their cultures, practices, goals, and communities.

@article{9b01452a52604a1e8841ae2da5cc3e57,
title = "Evolution and Revolution in Artificial Intelligence in Education",
abstract = "The field of Artificial Intelligence in Education (AIED) has undergone significant developments over the last twenty-five years. As we reflect on our past and shape our future, we ask two main questions: What are our major strengths? And, what new opportunities lay on the horizon? We analyse 47 papers from three years in the history of the Journal of AIED (1994, 2004, and 2014) to identify the foci and typical scenarios that occupy the field of AIED. We use those results to suggest two parallel strands of research that need to take place in order to impact education in the next 25 years: One is an evolutionary process, focusing on current classroom practices, collaborating with teachers, and diversifying technologies and domains. The other is a revolutionary process where we argue for embedding our technologies within students' everyday lives, supporting their cultures, practices, goals, and communities.",
keywords = "Artificial intelligence in education, Education revolution, Intelligent tutoring systems, Interactive learning environments",
author = "Ido Roll and Ruth Wylie",
note = "Publisher Copyright: {\textcopyright} 2016 International Artificial Intelligence in Education Society.",
year = "2016",
month = jun,
day = "1",
doi = "10.1007/s40593-016-0110-3",
language = "אנגלית",
volume = "26",
pages = "582--599",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "2",

}

Learning at scale 2016 preface

Aleven V, Haywood J, Kay J, Roll I. Learning at scale 2016 preface. L@S 2016 - Proceedings of the 3rd 2016 ACM Conference on Learning at Scale. 2016 Apr 25;iii-iv.
@article{8377d04ea9f644fb93f0e3cfee538506,
title = "Learning at scale 2016 preface",
author = "Vincent Aleven and Jeff Haywood and Judy Kay and Ido Roll",
year = "2016",
month = apr,
day = "25",
language = "אנגלית",
pages = "iii--iv",
journal = "L@S 2016 - Proceedings of the 3rd 2016 ACM Conference on Learning at Scale",
note = "3rd Annual ACM Conference on Learning at Scale, L@S 2016 ; Conference date: 25-04-2016 Through 26-04-2016",

}

Help Helps, but only so Much: Research on Help Seeking with Intelligent Tutoring Systems

Aleven V, Roll I, McLaren BM, Koedinger KR. Help Helps, but only so Much: Research on Help Seeking with Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education. 2016 Mar 1;26(1):205-223. https://doi.org/10.1007/s40593-015-0089-1
 

Help seeking is an important process in self-regulated learning (SRL). It may influence learning with intelligent tutoring systems (ITSs), because many ITSs provide help, often at the student's request. The Help Tutor was a tutor agent that gave in-context, real-time feedback on students' help-seeking behavior, as they were learning with an ITS. Key goals were to help students become better self-regulated learners and help them achieve better domain-level learning outcomes. In a classroom study, feedback on help seeking helped students to use on-demand help more deliberately, even after the feedback was no longer given, but not to achieve better learning outcomes. The work made a number of contributions, including the creation of a knowledge-engineered, rule-based, executable model of help seeking that can drive tutoring. We review these contributions from a contemporary perspective, with a theoretical analysis, a review of recent empirical literature on help seeking with ITSs, and methodological suggestions. Although we do not view on-demand, principle-based help during tutored problem solving as being as important as we once did, we still view it as helpful under certain circumstances, and recommend that it be included in ITSs. We view the goal of helping students become better self-regulated learners as one of the grand challenges in ITSs research today.

@article{3026ee7197c249f084001b2e5c3cc533,
title = "Help Helps, but only so Much: Research on Help Seeking with Intelligent Tutoring Systems",
abstract = "Help seeking is an important process in self-regulated learning (SRL). It may influence learning with intelligent tutoring systems (ITSs), because many ITSs provide help, often at the student's request. The Help Tutor was a tutor agent that gave in-context, real-time feedback on students' help-seeking behavior, as they were learning with an ITS. Key goals were to help students become better self-regulated learners and help them achieve better domain-level learning outcomes. In a classroom study, feedback on help seeking helped students to use on-demand help more deliberately, even after the feedback was no longer given, but not to achieve better learning outcomes. The work made a number of contributions, including the creation of a knowledge-engineered, rule-based, executable model of help seeking that can drive tutoring. We review these contributions from a contemporary perspective, with a theoretical analysis, a review of recent empirical literature on help seeking with ITSs, and methodological suggestions. Although we do not view on-demand, principle-based help during tutored problem solving as being as important as we once did, we still view it as helpful under certain circumstances, and recommend that it be included in ITSs. We view the goal of helping students become better self-regulated learners as one of the grand challenges in ITSs research today.",
keywords = "Classroom evaluation, Cognitive modeling, Help seeking, Intelligent tutoring systems, Metacognition, Self-regulated learning",
author = "Vincent Aleven and Ido Roll and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
note = "Publisher Copyright: {\textcopyright} 2016 International Artificial Intelligence in Education Society.",
year = "2016",
month = mar,
day = "1",
doi = "10.1007/s40593-015-0089-1",
language = "אנגלית",
volume = "26",
pages = "205--223",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "1",

}

Active learning in pre-class assignments: exploring the use of interactive simulations to enhance reading assignments

Stang JB, Barker M, Perez S, Ives J, Roll I. Active learning in pre-class assignments: exploring the use of interactive simulations to enhance reading assignments. Physics Education Research Conference Proceedings. 2016;332-335. https://doi.org/10.1119/perec.2016.pr.078
@article{64eaf048c12744b88bdf344ef2e0611e,
title = "Active learning in pre-class assignments: exploring the use of interactive simulations to enhance reading assignments",
author = "Stang, {Jared B.} and Megan Barker and Sarah Perez and Joss Ives and Ido Roll",
year = "2016",
doi = "10.1119/perec.2016.pr.078",
language = "אנגלית",
pages = "332--335",
journal = "Physics Education Research Conference Proceedings",
issn = "1539-9028",

}

An investigation of textbook-style highlighting for video

Fong M, Miller G, Zhang X, Roll I, Hendricks C, Fels S. An investigation of textbook-style highlighting for video. In Moffatt K, Popa T, editors, Graphics Interface 2016, GI 2016 - Proceedings. Canadian Information Processing Society. 2016. p. 201-208. (Proceedings - Graphics Interface).
 

Video is used extensively as an instructional aid within educational contexts such as blended (flipped) courses, self-learning with MOOCs and informal learning through online tutorials. One challenge is providing mechanisms for students to manage their video collection and quickly review or search for content. We provided students with a number of video interface features to establish which they would find most useful for video courses. From this, we designed an interface which uses textbook-style highlighting on a video filmstrip and transcript, both presented adjacent to a video player. This interface was qualitatively evaluated to determine if highlighting works well for saving intervals, and what strategies students use when given both direct video highlighting and the textbased transcript interface. Our participants reported that highlighting is a useful addition to instructional video. The familiar interaction of highlighting text was preferred, with the filmstrip used for intervals with more visual stimuli.

@inproceedings{31977437124f4f6a995ff6fe14811424,
title = "An investigation of textbook-style highlighting for video",
abstract = "Video is used extensively as an instructional aid within educational contexts such as blended (flipped) courses, self-learning with MOOCs and informal learning through online tutorials. One challenge is providing mechanisms for students to manage their video collection and quickly review or search for content. We provided students with a number of video interface features to establish which they would find most useful for video courses. From this, we designed an interface which uses textbook-style highlighting on a video filmstrip and transcript, both presented adjacent to a video player. This interface was qualitatively evaluated to determine if highlighting works well for saving intervals, and what strategies students use when given both direct video highlighting and the textbased transcript interface. Our participants reported that highlighting is a useful addition to instructional video. The familiar interaction of highlighting text was preferred, with the filmstrip used for intervals with more visual stimuli.",
keywords = "H.1.2. [models and principles]: User/machine systems, H.5.2. [information interfaces and presentation]: User interfaces",
author = "Matthew Fong and Gregor Miller and Xueqin Zhang and Ido Roll and Christina Hendricks and Sidney Fels",
year = "2016",
language = "אנגלית",
series = "Proceedings - Graphics Interface",
publisher = "Canadian Information Processing Society",
pages = "201--208",
editor = "Karyn Moffatt and Tiberiu Popa",
booktitle = "Graphics Interface 2016, GI 2016 - Proceedings",
address = "קנדה",
note = "42nd Graphics Interface 2016, GI 2016 ; Conference date: 01-06-2016 Through 03-06-2016",

}

2015

Evaluating the relationship between course structure, learner activity, and perceived value of online courses

Roll I, Macfadyen LP, Sandilands D. Evaluating the relationship between course structure, learner activity, and perceived value of online courses. In L@S 2015 - 2nd ACM Conference on Learning at Scale. Association for Computing Machinery. 2015. p. 385-388. (L@S 2015 - 2nd ACM Conference on Learning at Scale). https://doi.org/10.1145/2724660.2728699
 

Using aggregated Learning Management System data and course evaluation data from 26 online courses, we evaluated the relationship between measures of online activity, course and assessment structure, and student perceptions of course value. We find relationships between selected dimensions of learner engagement that reflect current constructivist theories of learning. This work demonstrates the potential value of pooled, easily accessible, and anonymous data for high-level inferences regarding design of online courses and the learner experience.

@inproceedings{ad8b592ffe9344268584dcc09554c974,
title = "Evaluating the relationship between course structure, learner activity, and perceived value of online courses",
abstract = "Using aggregated Learning Management System data and course evaluation data from 26 online courses, we evaluated the relationship between measures of online activity, course and assessment structure, and student perceptions of course value. We find relationships between selected dimensions of learner engagement that reflect current constructivist theories of learning. This work demonstrates the potential value of pooled, easily accessible, and anonymous data for high-level inferences regarding design of online courses and the learner experience.",
keywords = "Engagement, Learning analytics, Learning management system, Student evaluation of teaching",
author = "Ido Roll and Macfadyen, {Leah P.} and Debra Sandilands",
note = "Publisher Copyright: Copyright {\textcopyright} 2015 ACM.; 2nd ACM Conference on Learning at Scale, L@S 2015 ; Conference date: 14-03-2015 Through 18-03-2015",
year = "2015",
month = mar,
day = "14",
doi = "10.1145/2724660.2728699",
language = "אנגלית",
series = "L@S 2015 - 2nd ACM Conference on Learning at Scale",
publisher = "Association for Computing Machinery",
pages = "385--388",
booktitle = "L@S 2015 - 2nd ACM Conference on Learning at Scale",

}

Comparing representations for learner models in interactive simulations

Conati C, Fratamico L, Kardan S, Roll I. Comparing representations for learner models in interactive simulations. In Conati C, Heffernan N, Mitrovic A, Felisa Verdejo M, editors, Artificial Intelligence in Education - 17th International Conference, AIED 2015, Proceedings. Springer Verlag. 2015. p. 74-83. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-19773-9_8
 

Providing adaptive support in Exploratory Learning Environments is necessary but challenging due to the unstructured nature of interactions. This is especially the case for complex simulations such as the DC Circuit Construction Kit used in this work. To deal with this complexity, we evaluate alternative representations that capture different levels of detail in student interactions. Our results show that these representations can be effectively used in the user modeling framework proposed in [2], including behavior discovery and user classification, for student assessment and providing real-time support. We discuss trade-offs between high and low levels of detail in the tested interaction representations in terms of their ability to evaluate learning and inform feedback.

@inproceedings{81f955cc6ef243fd91d48020a816946e,
title = "Comparing representations for learner models in interactive simulations",
abstract = "Providing adaptive support in Exploratory Learning Environments is necessary but challenging due to the unstructured nature of interactions. This is especially the case for complex simulations such as the DC Circuit Construction Kit used in this work. To deal with this complexity, we evaluate alternative representations that capture different levels of detail in student interactions. Our results show that these representations can be effectively used in the user modeling framework proposed in [2], including behavior discovery and user classification, for student assessment and providing real-time support. We discuss trade-offs between high and low levels of detail in the tested interaction representations in terms of their ability to evaluate learning and inform feedback.",
keywords = "Clustering, Educational data mining, Exploratory learning environments, Interactive simulations, User modeling",
author = "Cristina Conati and Lauren Fratamico and Samad Kardan and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2015.; 17th International Conference on Artificial Intelligence in Education, AIED 2015 ; Conference date: 22-06-2015 Through 26-06-2015",
year = "2015",
doi = "10.1007/978-3-319-19773-9_8",
language = "אנגלית",
isbn = "9783319197722",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "74--83",
editor = "Cristina Conati and Neil Heffernan and Antonija Mitrovic and {Felisa Verdejo}, M.",
booktitle = "Artificial Intelligence in Education - 17th International Conference, AIED 2015, Proceedings",

}

Understanding, evaluating, and supporting self‐regulated learning using learning analytics

Roll I, Winne P. Understanding, evaluating, and supporting self‐regulated learning using learning analytics. Journal of Learning Analytics. 2015;2:7-12. https://doi.org/10.18608/jla.2015.21.2
@article{135a82c076c6438b91c87324820b5726,
title = "Understanding, evaluating, and supporting self‐regulated learning using learning analytics",
author = "Ido Roll and Philip Winne",
year = "2015",
doi = "10.18608/jla.2015.21.2",
language = "???core.languages.und???",
volume = "2",
pages = "7--12",
journal = "Journal of Learning Analytics",
issn = "1929-7750",
publisher = "UTS ePress",

}

2014

On the Benefits of Seeking (and Avoiding) Help in Online Problem-Solving Environments

Roll I, Baker RSJD, Aleven V, Koedinger KR. On the Benefits of Seeking (and Avoiding) Help in Online Problem-Solving Environments. Journal of the Learning Sciences. 2014 Oct 2;23(4):537-560. https://doi.org/10.1080/10508406.2014.883977
 

Seeking the right level of help at the right time can support learning. However, in the context of online problem-solving environments, it is still not entirely clear which help-seeking strategies are desired. We use fine-grained data from 38 high school students who worked with the Geometry Cognitive Tutor for 2 months to better understand the associations between specific help-seeking patterns and learning. We evaluate how students’ help-seeking behaviors on each step in a tutored problem are associated with their success on subsequent steps that require the same skills. Analyzing learning at the skill level allows us to compare different help-seeking patterns within a single student, controlling for between-student variations. Overall, asking for help on challenging steps is associated with productive learning, and overusing help is associated with poorer learning. However, contrary to many help-seeking theories, avoiding help (and failing repeatedly) is associated with better learning than seeking help on steps for which students have low prior knowledge. These results suggest that novice learners may benefit from engaging in solution attempts before they can make sense of given assistance. Methodological benefits for using local measures of learning are discussed, and comparisons are drawn to other forms of productive failure in problem solving.

@article{6c1831e0359b415e855c875c2f86c25e,
title = "On the Benefits of Seeking (and Avoiding) Help in Online Problem-Solving Environments",
abstract = "Seeking the right level of help at the right time can support learning. However, in the context of online problem-solving environments, it is still not entirely clear which help-seeking strategies are desired. We use fine-grained data from 38 high school students who worked with the Geometry Cognitive Tutor for 2 months to better understand the associations between specific help-seeking patterns and learning. We evaluate how students{\textquoteright} help-seeking behaviors on each step in a tutored problem are associated with their success on subsequent steps that require the same skills. Analyzing learning at the skill level allows us to compare different help-seeking patterns within a single student, controlling for between-student variations. Overall, asking for help on challenging steps is associated with productive learning, and overusing help is associated with poorer learning. However, contrary to many help-seeking theories, avoiding help (and failing repeatedly) is associated with better learning than seeking help on steps for which students have low prior knowledge. These results suggest that novice learners may benefit from engaging in solution attempts before they can make sense of given assistance. Methodological benefits for using local measures of learning are discussed, and comparisons are drawn to other forms of productive failure in problem solving.",
author = "Ido Roll and Baker, {Ryan S.J.d.} and Vincent Aleven and Koedinger, {Kenneth R.}",
note = "Publisher Copyright: {\textcopyright} , Copyright {\textcopyright} Taylor & Francis Group, LLC.",
year = "2014",
month = oct,
day = "2",
doi = "10.1080/10508406.2014.883977",
language = "אנגלית",
volume = "23",
pages = "537--560",
journal = "Journal of the Learning Sciences",
issn = "1050-8406",
publisher = "Routledge",
number = "4",

}

Interactions between teaching assistants and students boost engagement in physics labs

Stang JB, Roll I. Interactions between teaching assistants and students boost engagement in physics labs. Physical Review Special Topics - Physics Education Research. 2014 Sep 5;10(2):020117. https://doi.org/10.1103/PhysRevSTPER.10.020117
 

Through in-class observations of teaching assistants (TAs) and students in the lab sections of a large introductory physics course, we study which TA behaviors can be used to predict student engagement and, in turn, how this engagement relates to learning. For the TAs, we record data to determine how they adhere to and deliver the lesson plan and how they interact with students during the lab. For the students, we use observations to record the level of student engagement and pretests and post-tests of lab skills to measure learning. We find that the frequency of TA-student interactions, especially those initiated by the TAs, is a positive and significant predictor of student engagement. Interestingly, the length of interactions is not significantly correlated with student engagement. In addition, we find that student engagement was a better predictor of post-test performance than pretest scores. These results shed light on the manner in which students learn how to conduct inquiry and suggest that, by proactively engaging students, TAs may have a positive effect on student engagement, and therefore learning, in the lab.

@article{75fd8fe0c0864328816b3c2344dec9cd,
title = "Interactions between teaching assistants and students boost engagement in physics labs",
abstract = "Through in-class observations of teaching assistants (TAs) and students in the lab sections of a large introductory physics course, we study which TA behaviors can be used to predict student engagement and, in turn, how this engagement relates to learning. For the TAs, we record data to determine how they adhere to and deliver the lesson plan and how they interact with students during the lab. For the students, we use observations to record the level of student engagement and pretests and post-tests of lab skills to measure learning. We find that the frequency of TA-student interactions, especially those initiated by the TAs, is a positive and significant predictor of student engagement. Interestingly, the length of interactions is not significantly correlated with student engagement. In addition, we find that student engagement was a better predictor of post-test performance than pretest scores. These results shed light on the manner in which students learn how to conduct inquiry and suggest that, by proactively engaging students, TAs may have a positive effect on student engagement, and therefore learning, in the lab.",
author = "Stang, {Jared B.} and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Published by the American Physical Society.",
year = "2014",
month = sep,
day = "5",
doi = "10.1103/PhysRevSTPER.10.020117",
language = "אנגלית",
volume = "10",
journal = "Physical Review Special Topics - Physics Education Research",
issn = "1554-9178",
publisher = "American Physical Society",
number = "2",

}

Making the failure more productive: scaffolding the invention process to improve inquiry behaviors and outcomes in invention activities

Holmes NG, Day J, Park AHK, Bonn DA, Roll I. Making the failure more productive: scaffolding the invention process to improve inquiry behaviors and outcomes in invention activities. Instructional Science. 2014 Jul 1;42(4):523-538. https://doi.org/10.1007/s11251-013-9300-7
 

Invention activities are Productive Failure activities in which students attempt (and often fail) to invent methods that capture deep properties of a construct before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. While socio-cognitive theories of learning advocate for scaffolding in inquiry activities, reducing students’ agency, and possibly their failure rate, may be counter-productive in this context. Two Invention activities related to data analysis concepts were given to 87 undergraduate students in a first-year physics lab course using an interactive learning environment. Guided Invention students outperformed Unguided Invention students on measures of conceptual understanding of the structures of the constructs in an assessment two months after the learning period. There was no effect, however, on measures of procedural knowledge or conceptual understanding of the overall goals of the constructs. In addition, Guided Invention students were more likely to invent multiple methods during the Invention process. These results suggest that the domain-general scaffolding in Invention activities, when followed by instruction, can help students encode deep features of the domain and build on their failures during Productive Failure. These results further suggest not all failures are equally productive, and that some forms of support help students learn form their failed attempts.

@article{78a6f876107541859b99e1324a8f295c,
title = "Making the failure more productive: scaffolding the invention process to improve inquiry behaviors and outcomes in invention activities",
abstract = "Invention activities are Productive Failure activities in which students attempt (and often fail) to invent methods that capture deep properties of a construct before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. While socio-cognitive theories of learning advocate for scaffolding in inquiry activities, reducing students{\textquoteright} agency, and possibly their failure rate, may be counter-productive in this context. Two Invention activities related to data analysis concepts were given to 87 undergraduate students in a first-year physics lab course using an interactive learning environment. Guided Invention students outperformed Unguided Invention students on measures of conceptual understanding of the structures of the constructs in an assessment two months after the learning period. There was no effect, however, on measures of procedural knowledge or conceptual understanding of the overall goals of the constructs. In addition, Guided Invention students were more likely to invent multiple methods during the Invention process. These results suggest that the domain-general scaffolding in Invention activities, when followed by instruction, can help students encode deep features of the domain and build on their failures during Productive Failure. These results further suggest not all failures are equally productive, and that some forms of support help students learn form their failed attempts.",
keywords = "Interactive learning environments, Invention activities, Productive Failure, Scaffolding",
author = "Holmes, {N. G.} and James Day and Park, {Anthony H.K.} and Bonn, {D. A.} and Ido Roll",
note = "Publisher Copyright: {\textcopyright} 2013, Springer Science+Business Media Dordrecht.",
year = "2014",
month = jul,
day = "1",
doi = "10.1007/s11251-013-9300-7",
language = "אנגלית",
volume = "42",
pages = "523--538",
journal = "Instructional Science",
issn = "0020-4277",
publisher = "Springer Netherlands",
number = "4",

}

The usefulness of log based clustering in a complex simulation environment

Kardan S, Roll I, Conati C. The usefulness of log based clustering in a complex simulation environment. In Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Springer Verlag. 2014. p. 168-177. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-07221-0_21
 

Data mining techniques have been successfully employed on user interaction data in exploratory learning environments. In this paper we investigate using data mining techniques for analyzing student behaviors in an especially-complex exploratory environment, with over one hundred possible actions at any given point. Furthermore, the outcomes of these actions depend on their context. We propose a multi-layer action-events structure to deal with the complexity of the data and employ clustering and rule mining to examine student behaviors in terms of learning performance and effects of different degrees of scaffolding. Our findings show that using the proposed multi-layer structure for describing action-events enables the clustering algorithm to effectively identify the successful and unsuccessful students in terms of learning performance across activities in the presence or absence of external scaffolding. We also report and discuss the prominent behavior patterns of each group and investigate short term effects of scaffolding.

@inproceedings{a44b366773e84eab923bb8fb05bb2ec3,
title = "The usefulness of log based clustering in a complex simulation environment",
abstract = "Data mining techniques have been successfully employed on user interaction data in exploratory learning environments. In this paper we investigate using data mining techniques for analyzing student behaviors in an especially-complex exploratory environment, with over one hundred possible actions at any given point. Furthermore, the outcomes of these actions depend on their context. We propose a multi-layer action-events structure to deal with the complexity of the data and employ clustering and rule mining to examine student behaviors in terms of learning performance and effects of different degrees of scaffolding. Our findings show that using the proposed multi-layer structure for describing action-events enables the clustering algorithm to effectively identify the successful and unsuccessful students in terms of learning performance across activities in the presence or absence of external scaffolding. We also report and discuss the prominent behavior patterns of each group and investigate short term effects of scaffolding.",
keywords = "Clustering, Educational Data Mining, Scaffolding",
author = "Samad Kardan and Ido Roll and Cristina Conati",
year = "2014",
doi = "10.1007/978-3-319-07221-0_21",
language = "אנגלית",
isbn = "9783319072203",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "168--177",
booktitle = "Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings",
note = "12th International Conference on Intelligent Tutoring Systems, ITS 2014 ; Conference date: 05-06-2014 Through 09-06-2014",

}

A" flipped" approach to large-scale first-year labs

Rieger G, Sitwell M, Carolan J, Roll I. A" flipped" approach to large-scale first-year labs. Physics in Canada. 2014.
 
We describe a new approach to our first-year lab that has an eight-week formative
learning part followed by a summative application in a four-week project. A key feature
is that students perform some experiments at home and bring the data to class for
discussion and analysis. Performance on our diagnostic test shows that students are
generally learning the specific scientific skills that are targeted in our labs.
@article{c32bcac0d21941a9b96f41f4a521c475,
title = "A{"} flipped{"} approach to large-scale first-year labs",
abstract = "We describe a new approach to our first-year lab that has an eight-week formativelearning part followed by a summative application in a four-week project. A key featureis that students perform some experiments at home and bring the data to class fordiscussion and analysis. Performance on our diagnostic test shows that students aregenerally learning the specific scientific skills that are targeted in our labs.",
author = "Georg Rieger and Michael Sitwell and James Carolan and Ido Roll",
year = "2014",
language = "אנגלית",
journal = "Physics in Canada",

}

Enhancing self-regulated learning through metacognitively-aware intelligent tutoring systems

Goldberg B, Sottilare R, Roll I, Lajoie S, Poitras E, Biswas G et al. Enhancing self-regulated learning through metacognitively-aware intelligent tutoring systems. Proceedings of International Conference of the Learning Sciences, ICLS. 2014;3(January):1352-1361.
 

This symposium identifies current trends and future directions in research on metacognition and Self-Regulated Learning (SRL) in educational technologies, and specifically, Intelligent Tutoring Systems (ITS). Each paper will elaborate on detection and assessment of metacognition/SRL, forms of support and scaffolding, and self-and coregulation processes and authoring of environments that support ITS. The symposium will conclude with discussions that describe the manner in which metacognitive development can be promoted through strategies that support individual differences in multiple contexts. The alternative perspectives presented in this session will help advance our understanding of support for metacognition and SRL in ITS, as well as identify gaps that will influence future research pursuits.

@article{0a824a201baa4b6b8e139c82784a33cf,
title = "Enhancing self-regulated learning through metacognitively-aware intelligent tutoring systems",
abstract = "This symposium identifies current trends and future directions in research on metacognition and Self-Regulated Learning (SRL) in educational technologies, and specifically, Intelligent Tutoring Systems (ITS). Each paper will elaborate on detection and assessment of metacognition/SRL, forms of support and scaffolding, and self-and coregulation processes and authoring of environments that support ITS. The symposium will conclude with discussions that describe the manner in which metacognitive development can be promoted through strategies that support individual differences in multiple contexts. The alternative perspectives presented in this session will help advance our understanding of support for metacognition and SRL in ITS, as well as identify gaps that will influence future research pursuits.",
author = "Benjamin Goldberg and Robert Sottilare and Ido Roll and Susanne Lajoie and Eric Poitras and Gautam Biswas and Segedy, {James R.} and Kinnebrew, {John S.} and Wiese, {Eliane Stampfer} and Yanjin Long and Vincent Aleven and Koedinger, {Kenneth R.} and Phil Winne",
note = "Publisher Copyright: {\textcopyright} ISLS.; 11th International Conference of the Learning Sciences: Learning and Becoming in Practice, ICLS 2014 ; Conference date: 23-06-2014 Through 27-06-2014",
year = "2014",
language = "אנגלית",
volume = "3",
pages = "1352--1361",
journal = "Proceedings of International Conference of the Learning Sciences, ICLS",
issn = "1814-9316",
publisher = "International Society of the Learning Sciences",
number = "January",

}

Not a magic bullet: The effect of scaffolding on knowledge and attitudes in online simulations

Roll I, Briseno A, Yee N, Welsh A. Not a magic bullet: The effect of scaffolding on knowledge and attitudes in online simulations. Proceedings of International Conference of the Learning Sciences, ICLS. 2014;2(January):879-886.
 

Common wisdom and prior research suggest that students with low prior knowledge are in greater need for scaffolding. However, some forms of scaffolding may overload novice-students' cognitive capacity or short-circuit productive exploration of the problem space. Hence, we evaluate the effectiveness of scaffolding in a virtual simulation in physics, considering students' attitudes and prior knowledge. 100 undergraduate students completed either a scaffolded or a relatively unstructured activity, followed by another unstructured activity. While the given scaffolding was beneficial for students with high prior knowledge, it did not assist students with low prior knowledge. Furthermore, the scaffolded activity increased students' attitudes towards memory-and-recall in a way that transferred to the later unsupported activity, where these goals were no longer appropriate. Last, prior knowledge did not contribute to learning outcomes in the presence of intuitive grounded feedback. We explain these results in terms of productive failure and cognitive load.

@article{3709b845ec614386bd210e01bb014d64,
title = "Not a magic bullet: The effect of scaffolding on knowledge and attitudes in online simulations",
abstract = "Common wisdom and prior research suggest that students with low prior knowledge are in greater need for scaffolding. However, some forms of scaffolding may overload novice-students' cognitive capacity or short-circuit productive exploration of the problem space. Hence, we evaluate the effectiveness of scaffolding in a virtual simulation in physics, considering students' attitudes and prior knowledge. 100 undergraduate students completed either a scaffolded or a relatively unstructured activity, followed by another unstructured activity. While the given scaffolding was beneficial for students with high prior knowledge, it did not assist students with low prior knowledge. Furthermore, the scaffolded activity increased students' attitudes towards memory-and-recall in a way that transferred to the later unsupported activity, where these goals were no longer appropriate. Last, prior knowledge did not contribute to learning outcomes in the presence of intuitive grounded feedback. We explain these results in terms of productive failure and cognitive load.",
author = "Ido Roll and Adriana Briseno and Nikki Yee and Ashley Welsh",
note = "Publisher Copyright: {\textcopyright} ISLS.; 11th International Conference of the Learning Sciences: Learning and Becoming in Practice, ICLS 2014 ; Conference date: 23-06-2014 Through 27-06-2014",
year = "2014",
language = "אנגלית",
volume = "2",
pages = "879--886",
journal = "Proceedings of International Conference of the Learning Sciences, ICLS",
issn = "1814-9316",
publisher = "International Society of the Learning Sciences",
number = "January",

}

Participating in the Physics Lab: Does Gender Matter?

Holmes NG, Roll I, Bonn DA. Participating in the Physics Lab: Does Gender Matter? 2014.
@misc{8b0d1a3d88d94cff89e7b4ac5e083a5f,
title = "Participating in the Physics Lab: Does Gender Matter?",
author = "Holmes, {N. G.} and Ido Roll and Bonn, {D. A.}",
year = "2014",
language = "אנגלית",
type = "Other",

}

Students' adaptation and transfer of strategies across levels of scaffolding in an exploratory environment

Roll I, Yee N, Briseno A. Students' adaptation and transfer of strategies across levels of scaffolding in an exploratory environment. In Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings. Springer Verlag. 2014. p. 348-353. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-07221-0_43
 

While the effect of scaffolding on learning has received much attention, less is known about its effect on students' strategy use, especially in transfer activities. This study focuses on students' adaptive behaviours as a function of given scaffolding and when transitioning from a scaffolded to an unstructured activity. We study this in the context of a complex physics simulation in which students choose between 124 different actions. We evaluate (i) how the scaffolding affects students' building and testing behaviours, (ii) whether these behaviours transfer to an unstructured activity, and (iii) the relationship between the adapted behaviours and learning. A repeated-measures MANOVA suggests that students adapt their learning behaviours according to the demands and affordances of the task and the environment, and that these strategies transfer from a scaffolded to an unstructured activity. No significant relationships were found between these patterns and learning.

@inproceedings{58974ac5eab146ffa2423e424f74262f,
title = "Students' adaptation and transfer of strategies across levels of scaffolding in an exploratory environment",
abstract = "While the effect of scaffolding on learning has received much attention, less is known about its effect on students' strategy use, especially in transfer activities. This study focuses on students' adaptive behaviours as a function of given scaffolding and when transitioning from a scaffolded to an unstructured activity. We study this in the context of a complex physics simulation in which students choose between 124 different actions. We evaluate (i) how the scaffolding affects students' building and testing behaviours, (ii) whether these behaviours transfer to an unstructured activity, and (iii) the relationship between the adapted behaviours and learning. A repeated-measures MANOVA suggests that students adapt their learning behaviours according to the demands and affordances of the task and the environment, and that these strategies transfer from a scaffolded to an unstructured activity. No significant relationships were found between these patterns and learning.",
keywords = "inquiry learning, interactive simulations, microworlds, scaffolding, self-regulated learning, transfer",
author = "Ido Roll and Nikki Yee and Adriana Briseno",
year = "2014",
doi = "10.1007/978-3-319-07221-0_43",
language = "אנגלית",
isbn = "9783319072203",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "348--353",
booktitle = "Intelligent Tutoring Systems - 12th International Conference, ITS 2014, Proceedings",
note = "12th International Conference on Intelligent Tutoring Systems, ITS 2014 ; Conference date: 05-06-2014 Through 09-06-2014",

}

Tutoring self- and co-regulation with intelligent tutoring systems to help students acquire better learning skill

Sottilare R, (ed.), Graesser A, (ed.), Hu X, (ed.), Goldberg B, (ed.), Roll I, Wiese ES et al. Tutoring self- and co-regulation with intelligent tutoring systems to help students acquire better learning skill. In Sottilare R, Graesser A, Hu X, Goldberg B, editors, Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management. U.S. Army Research Laboratory. 2014. p. 169-182. (Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management).
 
A number of studies have found that students who better regulate their learning also achieve better
@inbook{30669538d24649e6866389a2ba6d5637,
title = "Tutoring self- and co-regulation with intelligent tutoring systems to help students acquire better learning skill",
abstract = "A number of studies have found that students who better regulate their learning also achieve better",
keywords = "design recommendations",
author = "Robert Sottilare and Arthur Graesser and Xiangen Hu and Benjamin Goldberg and Ido Roll and Wiese, {Eliane Stampfer} and Yanjin Long and Vincent Aleven and Koedinger, {Kenneth R.}",
year = "2014",
language = "American English",
isbn = "9780989392327",
series = "Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management",
publisher = "U.S. Army Research Laboratory",
pages = "169--182",
editor = "Robert Sottilare and Arthur Graesser and Xiangen Hu and Benjamin Goldberg",
booktitle = "Design Recommendations for Intelligent Tutoring Systems - Volume 2: Instructional Management",

}

2013

Finding Evidence Of Transfer With Invention Activities: Teaching The Concept Of Weighted Average

Day J, Holmes NG, Roll I, Bonn DA. Finding Evidence Of Transfer With Invention Activities: Teaching The Concept Of Weighted Average. Physics Education Research Conference Proceedings. 2013;117-120. https://doi.org/10.1119/perc.2013.pr.017
@article{d656de00599b4f44b122131bcbc5f1e8,
title = "Finding Evidence Of Transfer With Invention Activities: Teaching The Concept Of Weighted Average",
keywords = "invention activities, preparation for future learning, transfer, weighted average, measurement and uncertainty, Concise Data Processing Assessment",
author = "James Day and Holmes, {N. G.} and Ido Roll and Bonn, {D. A.}",
year = "2013",
doi = "10.1119/perc.2013.pr.017",
language = "אנגלית",
pages = "117--120",
journal = "Physics Education Research Conference Proceedings",
issn = "1539-9028",

}

Process and outcome benefits for orienting students to analyze and reflect on available data in productive failure activities

Roll I, Holmes NG, Day J, Park AHK, Bonn DA. Process and outcome benefits for orienting students to analyze and reflect on available data in productive failure activities. CEUR Workshop Proceedings. 2013;1009:61-68.
 

Invention activities are Productive Failure activities in which students at- tempt to invent methods that capture deep properties of given data before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. Two Invention activities related to data analysis concepts were given to 130 undergraduate students in a first-year physics lab course using an interactive learning environment. Students in the Guided Invention condition were given prompts to analyze given data prior to inventing and reflect on their methods after inventing them. These students outperformed Unguided Invention students on delayed measures of transfer, but not on measures of conceptual or proce- dural knowledge. In addition, Guided Invention students were more likely to invent multiple methods, suggesting that they used better self-regulated learning strategies.

@article{240913b5f28b4be7a25de42909c48fdf,
title = "Process and outcome benefits for orienting students to analyze and reflect on available data in productive failure activities",
abstract = "Invention activities are Productive Failure activities in which students at- tempt to invent methods that capture deep properties of given data before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. Two Invention activities related to data analysis concepts were given to 130 undergraduate students in a first-year physics lab course using an interactive learning environment. Students in the Guided Invention condition were given prompts to analyze given data prior to inventing and reflect on their methods after inventing them. These students outperformed Unguided Invention students on delayed measures of transfer, but not on measures of conceptual or proce- dural knowledge. In addition, Guided Invention students were more likely to invent multiple methods, suggesting that they used better self-regulated learning strategies.",
keywords = "Interactive learning environments, Invention activities, Productive failure, Scaffolding, Transfer.",
author = "Ido Roll and Holmes, {Natasha G.} and James Day and Park, {Anthony H.K.} and Bonn, {D. A.}",
year = "2013",
language = "אנגלית",
volume = "1009",
pages = "61--68",
journal = "CEUR Workshop Proceedings",
issn = "1613-0073",
publisher = "CEUR-WS",
note = "Workshops at the 16th International Conference on Artificial Intelligence in Education, AIED 2013 ; Conference date: 09-07-2013 Through 13-07-2013",

}

2012

Learning to Think: Cognitive Mechanisms of Knowledge Transfer

Koedinger KR, Roll I. Learning to Think: Cognitive Mechanisms of Knowledge Transfer. In The Oxford Handbook of Thinking and Reasoning. Oxford University Press. 2012 https://doi.org/10.1093/oxfordhb/9780199734689.013.0040
 

Learning to think is about transfer. The scope of transfer is essentially a knowledge representation question. Experiences during learning can lead to alternative latent representations of the acquired knowledge, not all of which are equally useful. Productive learning facilitates a general representation that yields accurate behavior in a large variety of new situations, thus enabling transfer. This chapter explores two hypotheses. First, learning to think happens in pieces and these pieces, or knowledge components, are the basis of a mechanistic explanation of transfer. This hypothesis yields an instructional engineering prescription: that scientific methods of cognitive task analysis can be used to discover these knowledge components, and the resulting cognitive models can be used to redesign instruction so as to foster better transfer. The second hypothesis is that symbolic languages act as agents of transfer by focusing learning on abstract knowledge components that can enhance thinking across a wide variety of situations. The language of algebra is a prime example and we use it to illustrate (1) that cognitive task analysis can reveal knowledge components hidden to educators; (2) that such components may be acquired, like first language grammar rules, implicitly through practice; (3) that these components may be "big ideas" not in their complexity but in terms of their usefulness as they produce transfer across contexts; and (4) that domainspecific knowledge analysis is critical to effective application of domain-general instructional strategies.

@inbook{33e4804c6a0c4d96a13fa8fe40e6f8a7,
title = "Learning to Think: Cognitive Mechanisms of Knowledge Transfer",
abstract = "Learning to think is about transfer. The scope of transfer is essentially a knowledge representation question. Experiences during learning can lead to alternative latent representations of the acquired knowledge, not all of which are equally useful. Productive learning facilitates a general representation that yields accurate behavior in a large variety of new situations, thus enabling transfer. This chapter explores two hypotheses. First, learning to think happens in pieces and these pieces, or knowledge components, are the basis of a mechanistic explanation of transfer. This hypothesis yields an instructional engineering prescription: that scientific methods of cognitive task analysis can be used to discover these knowledge components, and the resulting cognitive models can be used to redesign instruction so as to foster better transfer. The second hypothesis is that symbolic languages act as agents of transfer by focusing learning on abstract knowledge components that can enhance thinking across a wide variety of situations. The language of algebra is a prime example and we use it to illustrate (1) that cognitive task analysis can reveal knowledge components hidden to educators; (2) that such components may be acquired, like first language grammar rules, implicitly through practice; (3) that these components may be {"}big ideas{"} not in their complexity but in terms of their usefulness as they produce transfer across contexts; and (4) that domainspecific knowledge analysis is critical to effective application of domain-general instructional strategies.",
keywords = "Cognitive task analysis, Cognitive tutors, Computational modeling, Educational technology, In vivo experiments, Language and math learning, Transfer",
author = "Koedinger, {Kenneth R.} and Ido Roll",
note = "Publisher Copyright: {\textcopyright} Oxford University Press, 2014.",
year = "2012",
month = nov,
day = "21",
doi = "10.1093/oxfordhb/9780199734689.013.0040",
language = "אנגלית",
isbn = "9780199734689",
booktitle = "The Oxford Handbook of Thinking and Reasoning",
publisher = "Oxford University Press",
address = "ארצות הברית",

}

Evaluating metacognitive scaffolding in Guided Invention Activities

Roll I, Holmes NG, Day J, Bonn D. Evaluating metacognitive scaffolding in Guided Invention Activities. Instructional Science. 2012 Jul;40(4):691-710. https://doi.org/10.1007/s11251-012-9208-7
 

Invention and Productive Failure activities ask students to generate methods that capture the important properties of some given data (e. g., uncertainty) before being taught the expert solution. Invention and Productive Failure activities are a class of scientific inquiry activities in that students create, implement, and evaluate mathematical models based on data. Yet, lacking sufficient inquiry skills, students often do not actualize the full potential of these activities. We identified key invention strategies in which students often fail to engage: exploratory analysis, peer interaction, self-explanation, and evaluation. A classroom study with 134 students evaluated the effect of supporting these skills on the quality and outcomes of the invention process. Students in the Unguided Invention condition received conventional Invention Activities; students in the Guided Invention condition received complementary metacognitive scaffolding. Students were asked to invent methods for calculating uncertainties in best-fitting lines. Guided Invention students invented methods that included more conceptual features and ranked the given datasets more accurately, although the quality of their mathematical expressions was not improved. At the process level, Guided Invention students revised their methods more frequently and had more and better instances of unprompted self-explanations even on components of the activity that were not supported by the metacognitive scaffolding. Classroom observations are used to demonstrate the effect of the scaffolding on students' learning behaviours. These results suggest that process guidance in the form of metacognitive scaffolding augments the inherent benefits of Invention Activities and can lead to gains at both domain and inquiry levels.

@article{a665682f57764b6bb1c35e73e51e2aa5,
title = "Evaluating metacognitive scaffolding in Guided Invention Activities",
abstract = "Invention and Productive Failure activities ask students to generate methods that capture the important properties of some given data (e. g., uncertainty) before being taught the expert solution. Invention and Productive Failure activities are a class of scientific inquiry activities in that students create, implement, and evaluate mathematical models based on data. Yet, lacking sufficient inquiry skills, students often do not actualize the full potential of these activities. We identified key invention strategies in which students often fail to engage: exploratory analysis, peer interaction, self-explanation, and evaluation. A classroom study with 134 students evaluated the effect of supporting these skills on the quality and outcomes of the invention process. Students in the Unguided Invention condition received conventional Invention Activities; students in the Guided Invention condition received complementary metacognitive scaffolding. Students were asked to invent methods for calculating uncertainties in best-fitting lines. Guided Invention students invented methods that included more conceptual features and ranked the given datasets more accurately, although the quality of their mathematical expressions was not improved. At the process level, Guided Invention students revised their methods more frequently and had more and better instances of unprompted self-explanations even on components of the activity that were not supported by the metacognitive scaffolding. Classroom observations are used to demonstrate the effect of the scaffolding on students' learning behaviours. These results suggest that process guidance in the form of metacognitive scaffolding augments the inherent benefits of Invention Activities and can lead to gains at both domain and inquiry levels.",
keywords = "Guided discovery, Inquiry learning, Invention Activities, Metacognitive scaffolding, Productive Failure",
author = "Ido Roll and Holmes, {Natasha G.} and James Day and Doug Bonn",
note = "Funding Information: Acknowledgments This work was supported by the Pittsburgh Science of Learning Center, which is supported by the National Science Foundation (#SBE-0836012), and by the University of British Columbia through the Carl Wieman Science Education Initiative.",
year = "2012",
month = jul,
doi = "10.1007/s11251-012-9208-7",
language = "אנגלית",
volume = "40",
pages = "691--710",
journal = "Instructional Science",
issn = "0020-4277",
publisher = "Springer Netherlands",
number = "4",

}

Progress in assessment and tutoring of lifelong learning skills: An intelligent tutor agent that helps students become better help seekers

Aleven V, Roll I, Koedinger KR. Progress in assessment and tutoring of lifelong learning skills: An intelligent tutor agent that helps students become better help seekers. In Adaptive Technologies for Training and Education. Cambridge University Press. 2012. p. 69-95 https://doi.org/10.1017/CBO9781139049580.008
 

Intelligent Tutoring Systems (ITSs) have been shown to enhance learning in a range of domains, including mathematics, physics, computer programming, electronics troubleshooting, database design, medical diagnosis, and others (Beal, Walles, Arroyo, & Woolf, 2007; Crowley et al., 2007; Gott & Lesgold, 2000; Graesser, Chipman, Haynes, & Olney, 2005; Koedinger & Aleven, 2007; Koedinger, Anderson, Hadley, & Mark, 1997; Martin & Mitrovic, 2002; Mitrovic et al., 2008; Mostow & Beck, 2007; Rickel & Johnson, 1999; VanLehn et al., 2005). In this chapter we take up the question whether ITSs can help learners foster lifelong learning skills. By this term we refer to skills and strategies that enable people to be effective learners in a range of domains. Domain-general learning skills are important “tools” for learners, because the formal schooling system cannot prepare students for all knowledge or skills they will ever need. Prior to the study reported in the current chapter, there was limited evidence that ITSs can support learners in acquiring lifelong learning skills. We feel we have interesting progress to report: We found evidence that an ITS can support students in becoming better at seeking help as they work with an ITS. Researchers have long studied the self-regulatory processes that effective learners exhibit in a range of learning environments, with and (primarily) without computers. This line of work has produced a number of comprehensive theoretical frameworks for self-regulated learning (Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman, 2008). Other work has focused on creating instructional interventions that emphasize self-regulatory or metacognitive aspects of learning, including successful classroom programs for: learning to read with understanding through reciprocal teaching (Palincsar & Brown, 1984); self-assessment and classroom discussion thereof related to a science inquiry cycle (White & Frederiksen, 1998); using self-addressed metacognitive questions in the domain of mathematics learning (Mevarech & Fridkin, 2006); and reflecting on quiz feedback in college-level remedial mathematics (Zimmerman & Moylan, 2009).

@inbook{7dd284963fcf478890ca6fc5ada05a11,
title = "Progress in assessment and tutoring of lifelong learning skills: An intelligent tutor agent that helps students become better help seekers",
abstract = "Intelligent Tutoring Systems (ITSs) have been shown to enhance learning in a range of domains, including mathematics, physics, computer programming, electronics troubleshooting, database design, medical diagnosis, and others (Beal, Walles, Arroyo, & Woolf, 2007; Crowley et al., 2007; Gott & Lesgold, 2000; Graesser, Chipman, Haynes, & Olney, 2005; Koedinger & Aleven, 2007; Koedinger, Anderson, Hadley, & Mark, 1997; Martin & Mitrovic, 2002; Mitrovic et al., 2008; Mostow & Beck, 2007; Rickel & Johnson, 1999; VanLehn et al., 2005). In this chapter we take up the question whether ITSs can help learners foster lifelong learning skills. By this term we refer to skills and strategies that enable people to be effective learners in a range of domains. Domain-general learning skills are important “tools” for learners, because the formal schooling system cannot prepare students for all knowledge or skills they will ever need. Prior to the study reported in the current chapter, there was limited evidence that ITSs can support learners in acquiring lifelong learning skills. We feel we have interesting progress to report: We found evidence that an ITS can support students in becoming better at seeking help as they work with an ITS. Researchers have long studied the self-regulatory processes that effective learners exhibit in a range of learning environments, with and (primarily) without computers. This line of work has produced a number of comprehensive theoretical frameworks for self-regulated learning (Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman, 2008). Other work has focused on creating instructional interventions that emphasize self-regulatory or metacognitive aspects of learning, including successful classroom programs for: learning to read with understanding through reciprocal teaching (Palincsar & Brown, 1984); self-assessment and classroom discussion thereof related to a science inquiry cycle (White & Frederiksen, 1998); using self-addressed metacognitive questions in the domain of mathematics learning (Mevarech & Fridkin, 2006); and reflecting on quiz feedback in college-level remedial mathematics (Zimmerman & Moylan, 2009).",
author = "Vincent Aleven and Ido Roll and Koedinger, {Kenneth R.}",
note = "Publisher Copyright: {\textcopyright} Cambridge University Press 2012.",
year = "2012",
month = jan,
day = "1",
doi = "10.1017/CBO9781139049580.008",
language = "אנגלית",
isbn = "9780521769037",
pages = "69--95",
booktitle = "Adaptive Technologies for Training and Education",
publisher = "Cambridge University Press",

}

Building (Timely) bridges between learning analytics, educational data mining and core learning sciences perspectives

Roll I, Aleven V, Koedinger KR, Berland M, Martin T, Benton T et al. Building (Timely) bridges between learning analytics, educational data mining and core learning sciences perspectives. In 10th International Conference of the Learning Sciences: The Future of Learning, ICLS 2012 - Proceedings. 2012. p. 134-141. (10th International Conference of the Learning Sciences: The Future of Learning, ICLS 2012 - Proceedings).
 

Despite the exponential growth of the research on Learning Analytics (LA) and Educational Data Mining (EDM) over the last few years, the work has been still distant from the core Learning Sciences methods, theoretical constructs, and literature. At the same time, over the last 15 years, Learning Sciences as a field has been quite innovative, eclectic, and effective in incorporating new methodological stances, such as micro-genetic methods, micro-ethnographies, and design-based research. It seems that the time has come to build sound connections between these traditions. The goal of this symposium is to bring together researchers coming from different academic perspectives, to explore and examine common LA/EDM methodological and theoretical threads with wide applicability within the Learning Sciences. The papers presented explore text mining in clinical interviews, moment-by-moment learning curves and traces, data mining of programming logs, and cognitive tutors, representing the main perspectives and methodological approaches in the field.

@inproceedings{83f28d7e3df24dffba47b390e1d559ee,
title = "Building (Timely) bridges between learning analytics, educational data mining and core learning sciences perspectives",
abstract = "Despite the exponential growth of the research on Learning Analytics (LA) and Educational Data Mining (EDM) over the last few years, the work has been still distant from the core Learning Sciences methods, theoretical constructs, and literature. At the same time, over the last 15 years, Learning Sciences as a field has been quite innovative, eclectic, and effective in incorporating new methodological stances, such as micro-genetic methods, micro-ethnographies, and design-based research. It seems that the time has come to build sound connections between these traditions. The goal of this symposium is to bring together researchers coming from different academic perspectives, to explore and examine common LA/EDM methodological and theoretical threads with wide applicability within the Learning Sciences. The papers presented explore text mining in clinical interviews, moment-by-moment learning curves and traces, data mining of programming logs, and cognitive tutors, representing the main perspectives and methodological approaches in the field.",
author = "Ido Roll and Vincent Aleven and Koedinger, {Kenneth R.} and Matthew Berland and Taylor Martin and Tom Benton and Carmen Petrick and Arnon Hershkovitz and Michael Wixon and Ryan Baker and Janice Gobert and Pedro, {Michael Sao} and Bruce Sherin and Paulo Blikstein and Marcelo Worsley and Roy Pea",
year = "2012",
language = "אנגלית",
isbn = "9780578107042",
series = "10th International Conference of the Learning Sciences: The Future of Learning, ICLS 2012 - Proceedings",
pages = "134--141",
booktitle = "10th International Conference of the Learning Sciences",
note = "10th International Conference of the Learning Sciences: The Future of Learning, ICLS 2012 ; Conference date: 02-07-2012 Through 06-07-2012",

}

2011

Improving students' help-seeking skills using metacognitive feedback in an intelligent tutoring system

Roll I, Aleven V, McLaren BM, Koedinger KR. Improving students' help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction. 2011 Apr;21(2):267-280. https://doi.org/10.1016/j.learninstruc.2010.07.004
 

The present research investigated whether immediate metacognitive feedback on students' help-seeking errors can help students acquire better help-seeking skills. The Help Tutor, an intelligent tutor agent for help seeking, was integrated into a commercial tutoring system for geometry, the Geometry Cognitive Tutor. Study 1, with 58 students, found that the real-time assessment of students' help-seeking behavior correlated with other independent measures of help seeking, and that the Help Tutor improved students' help-seeking behavior while learning Geometry with the Geometry Cognitive Tutor. Study 2, with 67 students, evaluated more elaborated support that included, in addition to the Help Tutor, also help-seeking instruction and support for self-assessment. The study replicated the effect found in Study 1. It was also found that the improved help-seeking skills transferred to learning new domain-level content during the month following the intervention, while the help-seeking support was no longer in effect. Implications for metacognitive tutoring are discussed.

@article{e9496243fa884c738c035776d5ddc8de,
title = "Improving students' help-seeking skills using metacognitive feedback in an intelligent tutoring system",
abstract = "The present research investigated whether immediate metacognitive feedback on students' help-seeking errors can help students acquire better help-seeking skills. The Help Tutor, an intelligent tutor agent for help seeking, was integrated into a commercial tutoring system for geometry, the Geometry Cognitive Tutor. Study 1, with 58 students, found that the real-time assessment of students' help-seeking behavior correlated with other independent measures of help seeking, and that the Help Tutor improved students' help-seeking behavior while learning Geometry with the Geometry Cognitive Tutor. Study 2, with 67 students, evaluated more elaborated support that included, in addition to the Help Tutor, also help-seeking instruction and support for self-assessment. The study replicated the effect found in Study 1. It was also found that the improved help-seeking skills transferred to learning new domain-level content during the month following the intervention, while the help-seeking support was no longer in effect. Implications for metacognitive tutoring are discussed.",
keywords = "Cognitive tutors, Help seeking, Intelligent tutoring systems, Metacognition, Self-regulated learning",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
note = "Funding Information: We thank Ryan Baker, Jo Bodnar, Ido Jamar, Terri Murphy, Sabine Lynn, Kris Hobaugh, Kathy Dickensheets, Grant McKinney, EJ Ryu, and Christy McGuire for their help. This work was supported by the Pittsburgh Science of Learning Center, which is supported by the National Science Foundation ( #SBE-0354420 ), by a Graduate Training Grant awarded by the Department of Education ( #R305B040063 ), and by a grant from the National Science Foundation ( #IIS-0308200 ). ",
year = "2011",
month = apr,
doi = "10.1016/j.learninstruc.2010.07.004",
language = "אנגלית",
volume = "21",
pages = "267--280",
journal = "Learning and Instruction",
issn = "0959-4752",
publisher = "Elsevier BV",
number = "2",

}

Metacognitive practice makes perfect: Improving students' self-assessment skills with an intelligent tutoring system

Roll I, Aleven V, McLaren BM, Koedinger KR. Metacognitive practice makes perfect: Improving students' self-assessment skills with an intelligent tutoring system. In Artificial Intelligence in Education - 15th International Conference, AIED 2011. 2011. p. 288-295. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-21869-9_38
 

Helping students' improve their metacognitive and self-regulation skills holds the potential to improve students' ability to learn independently. Yet, to date, there are relatively few success stories of helping students enhance their metacognitive skills using interactive learning environments. In this paper we describe the Self-Assessment Tutor, an intelligent tutoring system for improving the accuracy of the judgments students make regarding their own knowledge. A classroom evaluation of the Self-Assessment Tutor with 84 students found that students improved their ability to identify their strengths while working with the Self-Assessment Tutor. In addition, students transferred the improved self-assessment skills to corresponding sections in the Geometry Cognitive Tutor. However, students often failed to identify their knowledge deficits a-priori and failed to update their assessments following unsuccessful solution attempts. This study contributes to theories of Self-Assessment and provides support for the viability of improving metacognitive skills using intelligent tutoring systems.

@inproceedings{11b00dbcb45d463c992dc828edf21277,
title = "Metacognitive practice makes perfect: Improving students' self-assessment skills with an intelligent tutoring system",
abstract = "Helping students' improve their metacognitive and self-regulation skills holds the potential to improve students' ability to learn independently. Yet, to date, there are relatively few success stories of helping students enhance their metacognitive skills using interactive learning environments. In this paper we describe the Self-Assessment Tutor, an intelligent tutoring system for improving the accuracy of the judgments students make regarding their own knowledge. A classroom evaluation of the Self-Assessment Tutor with 84 students found that students improved their ability to identify their strengths while working with the Self-Assessment Tutor. In addition, students transferred the improved self-assessment skills to corresponding sections in the Geometry Cognitive Tutor. However, students often failed to identify their knowledge deficits a-priori and failed to update their assessments following unsuccessful solution attempts. This study contributes to theories of Self-Assessment and provides support for the viability of improving metacognitive skills using intelligent tutoring systems.",
keywords = "Metacognition, Self-assessment, feeling of knowing (FOK), help seeking, intelligent tutoring systems, self-regulated learning",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
year = "2011",
doi = "10.1007/978-3-642-21869-9_38",
language = "אנגלית",
isbn = "9783642218682",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "288--295",
booktitle = "Artificial Intelligence in Education - 15th International Conference, AIED 2011",
note = "15th International Conference on Artificial Intelligence in Education, AIED 2011 ; Conference date: 28-06-2011 Through 01-07-2011",

}

Outcomes and mechanisms of transfer in invention activities

Roll I, Aleven V, Koedinger K. Outcomes and mechanisms of transfer in invention activities. In Proceedings of the annual meeting of the cognitive science society. Vol. 33. 2011
@inproceedings{1f2a4044f7b74d31898dd35e98f80b8e,
title = "Outcomes and mechanisms of transfer in invention activities",
author = "Ido Roll and Vincent Aleven and Ken Koedinger",
year = "2011",
language = "American English",
volume = "33",
booktitle = "Proceedings of the annual meeting of the cognitive science society",

}

Outcomes and Mechanisms of Transfer in Invention Activities

Roll I, Aleven V, Koedinger KR. Outcomes and Mechanisms of Transfer in Invention Activities. In Carlson L, Hoelscher C, Shipley TF, editors, Expanding the Space of Cognitive Science - Proceedings of the 33rd Annual Meeting of the Cognitive Science Society, CogSci 2011. The Cognitive Science Society. 2011. p. 2824-2829. (Expanding the Space of Cognitive Science - Proceedings of the 33rd Annual Meeting of the Cognitive Science Society, CogSci 2011).
 

Invention activities are structured tasks in which students create mathematical methods that attempt to capture deep properties of data (e.g., variability), prior to receiving instruction on canonical methods (e.g., mean deviation). While experiments have demonstrated the learning benefits of invention activities, the mechanisms of transfer remain unknown. We address this question by evaluating the role of design in invention activities, identifying what knowledge is acquired during invention activities, and how it is applied in transfer tasks. A classroom experiment with 92 students compared the full invention process to one in which students evaluate predesigned methods. Results show that students in the full invention condition acquired more adaptive knowledge, yet not necessarily better procedural knowledge or invention skills. We suggest a mechanism that explains what knowledge invention attempts produce, how that knowledge is productively modified in subsequent instruction, and how it improves performance on some measures of transfer but not others.

@inproceedings{9fa404bf1e26476abbe1002681033c08,
title = "Outcomes and Mechanisms of Transfer in Invention Activities",
abstract = "Invention activities are structured tasks in which students create mathematical methods that attempt to capture deep properties of data (e.g., variability), prior to receiving instruction on canonical methods (e.g., mean deviation). While experiments have demonstrated the learning benefits of invention activities, the mechanisms of transfer remain unknown. We address this question by evaluating the role of design in invention activities, identifying what knowledge is acquired during invention activities, and how it is applied in transfer tasks. A classroom experiment with 92 students compared the full invention process to one in which students evaluate predesigned methods. Results show that students in the full invention condition acquired more adaptive knowledge, yet not necessarily better procedural knowledge or invention skills. We suggest a mechanism that explains what knowledge invention attempts produce, how that knowledge is productively modified in subsequent instruction, and how it improves performance on some measures of transfer but not others.",
keywords = "Invention activities, generation, intelligent tutoring systems, modular knowledge, transfer",
author = "Ido Roll and Vincent Aleven and Koedinger, {Kenneth R.}",
note = "Publisher Copyright: {\textcopyright} CogSci 2011.; 33rd Annual Meeting of the Cognitive Science Society: Expanding the Space of Cognitive Science, CogSci 2011 ; Conference date: 20-07-2011 Through 23-07-2011",
year = "2011",
language = "אנגלית",
series = "Expanding the Space of Cognitive Science - Proceedings of the 33rd Annual Meeting of the Cognitive Science Society, CogSci 2011",
publisher = "The Cognitive Science Society",
pages = "2824--2829",
editor = "Laura Carlson and Christoph Hoelscher and Shipley, {Thomas F.}",
booktitle = "Expanding the Space of Cognitive Science - Proceedings of the 33rd Annual Meeting of the Cognitive Science Society, CogSci 2011",

}

2010

Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system

Aleven V, Roll I, McLaren BM, Koedinger KR. Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist. 2010 Oct;45(4):224-233. https://doi.org/10.1080/00461520.2010.517740
 

Assessment of students' self-regulated learning (SRL) requires a method for evaluating whether observed actions are appropriate acts of self-regulation in the specific learning context in which they occur. We review research that has resulted in an automated method for context-sensitive assessment of a specific SRL strategy, help seeking while working with an intelligent tutoring system. The method relies on a computer-executable model of the targeted SRL strategy. The method was validated by showing that it converges with other measures of help seeking. Automated feedback on help seeking driven by this method led to a lasting improvement in students' help-seeking behavior, although not in domain-specific learning. The method is unobtrusive, is temporally fine-grained, and can be applied on a large scale and over extended periods. The approach could be applied to other SRL strategies besides help seeking.

@article{9c474daad5214da49fc5783954ac42ed,
title = "Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system",
abstract = "Assessment of students' self-regulated learning (SRL) requires a method for evaluating whether observed actions are appropriate acts of self-regulation in the specific learning context in which they occur. We review research that has resulted in an automated method for context-sensitive assessment of a specific SRL strategy, help seeking while working with an intelligent tutoring system. The method relies on a computer-executable model of the targeted SRL strategy. The method was validated by showing that it converges with other measures of help seeking. Automated feedback on help seeking driven by this method led to a lasting improvement in students' help-seeking behavior, although not in domain-specific learning. The method is unobtrusive, is temporally fine-grained, and can be applied on a large scale and over extended periods. The approach could be applied to other SRL strategies besides help seeking.",
author = "Vincent Aleven and Ido Roll and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
note = "Funding Information: The writing of this review article was sponsored by National Science Foundation Award SBE0354420 to the Pittsburgh Science of Learning Center.",
year = "2010",
month = oct,
doi = "10.1080/00461520.2010.517740",
language = "אנגלית",
volume = "45",
pages = "224--233",
journal = "Educational Psychologist",
issn = "0046-1520",
publisher = "Routledge",
number = "4",

}

Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors

Baker RSJD, Gobert JD, Van Joolingen W, Azevedo R, Roll I, São Pedro M et al.. Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors. 2010. Paper presented at 9th International Conference of the Learning Sciences, ICLS 2010, Chicago, IL, United States.
 

This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students' learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement.

@conference{62897ef139974d23a1ca23bb89316e9d,
title = "Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors",
abstract = "This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students' learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement.",
author = "Baker, {Ryan S.J.D.} and Gobert, {Janice D.} and {Van Joolingen}, Wouter and Roger Azevedo and Ido Roll and {S{\~a}o Pedro}, Michael and Juelaila Raziuddin and Nathan Krach and {De Carvalho}, {Adriana M.J.B.} and Jay Raspat and Vincent Aleven and Corbett, {Albert T.} and Koedinger, {Kenneth R.} and Mihaela Cocea and Arnon Hershkovitz and Amy Witherspoon and Amber Chauncey and Mihai Lintean and Zhiqiang Cai and Vasile Rus and Arthur Greesser",
year = "2010",
language = "אנגלית",
pages = "45--52",
note = "9th International Conference of the Learning Sciences, ICLS 2010 ; Conference date: 29-06-2010 Through 02-07-2010",

}

The invention Lab: Using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments

Roll I, Aleven V, Koedinger KR. The invention Lab: Using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments. In Intelligent Tutoring Systems - 10th International Conference, ITS 2010, Proceedings. PART 1 ed. 2010. p. 115-124. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). https://doi.org/10.1007/978-3-642-13388-6_16
 

Exploratory Learning Environments (ELE) facilitate scientific inquiry tasks in which learners attempt to develop or uncover underlying scientific or mathematical models. Unlike step-based Intelligent Tutoring Systems (ITS), and due to task characteristics and pedagogical philosophy, ELE offer little support at the domain level. Lacking adequate support, ELE often fail to deliver on their promise. We describe the Invention Lab, a system that combines the benefits of ELE and ITS by offering adaptive support in a relatively unconstrained environment. The Invention Lab combines modeling techniques to assess students' knowledge at the domain and inquiry levels. The system uses this information to design new tasks in real time, thus adapting to students' needs while maintaining critical features of the inquiry process. Data from an in-class evaluation study illustrates how the Invention Lab helps students develop sophisticated mathematical models and improve their scientific inquiry behavior. Implications for intelligent support in ELE are discussed.

@inproceedings{77de9206028e4e14b63c2c13ba38de24,
title = "The invention Lab: Using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments",
abstract = "Exploratory Learning Environments (ELE) facilitate scientific inquiry tasks in which learners attempt to develop or uncover underlying scientific or mathematical models. Unlike step-based Intelligent Tutoring Systems (ITS), and due to task characteristics and pedagogical philosophy, ELE offer little support at the domain level. Lacking adequate support, ELE often fail to deliver on their promise. We describe the Invention Lab, a system that combines the benefits of ELE and ITS by offering adaptive support in a relatively unconstrained environment. The Invention Lab combines modeling techniques to assess students' knowledge at the domain and inquiry levels. The system uses this information to design new tasks in real time, thus adapting to students' needs while maintaining critical features of the inquiry process. Data from an in-class evaluation study illustrates how the Invention Lab helps students develop sophisticated mathematical models and improve their scientific inquiry behavior. Implications for intelligent support in ELE are discussed.",
keywords = "Constraint-based modeling, Exploratory learning environments, Intelligent tutoring systems, Invention as preparation for learning, Model tracing",
author = "Ido Roll and Vincent Aleven and Koedinger, {Kenneth R.}",
year = "2010",
doi = "10.1007/978-3-642-13388-6_16",
language = "אנגלית",
isbn = "3642133878",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
number = "PART 1",
pages = "115--124",
booktitle = "Intelligent Tutoring Systems - 10th International Conference, ITS 2010, Proceedings",
edition = "PART 1",
note = "10th International Conference on Intelligent Tutoring Systems, ITS 2010 ; Conference date: 14-06-2010 Through 18-06-2010",

}

2009

Helping students know ‘further’ - increasing the flexibility of students’ knowledge using symbolic invention tasks

Roll I, Koedinger KR. Helping students know ‘further’ - increasing the flexibility of students’ knowledge using symbolic invention tasks. In Cognitive Science Society. 2009
@inproceedings{ca57de5554304aacb631cf3ead7982c8,
title = "Helping students know {\textquoteleft}further{\textquoteright} - increasing the flexibility of students{\textquoteright} knowledge using symbolic invention tasks",
author = "Ido Roll and Koedinger, {Kenneth R.}",
year = "2009",
language = "American English",
booktitle = "Cognitive Science Society",

}

2008

Developing a generalizable detector of when students game the system

Baker RSJD, Corbett AT, Roll I, Koedinger KR. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction. 2008 Aug;18(3):287-314. https://doi.org/10.1007/s11257-007-9045-6
 

Some students, when working in interactive learning environments, attempt to "game the system", attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector's generalizability, and find that it transfers successfully to both new students and new tutor lessons.

@article{172ae3288a124ed8bf4f0aec16f9c292,
title = "Developing a generalizable detector of when students game the system",
abstract = "Some students, when working in interactive learning environments, attempt to {"}game the system{"}, attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector's generalizability, and find that it transfers successfully to both new students and new tutor lessons.",
keywords = "Behavior detection, Cognitive tutors, Gaming the system, Generalizable models, Interactive learning environments, Latent response models, Machine learning, Student modeling",
author = "Baker, {Ryan S.J.D.} and Corbett, {Albert T.} and Ido Roll and Koedinger, {Kenneth R.}",
note = "Funding Information: Acknowledgements This research was supported by an NDSEG (National Defense Science and Engineering Graduate) Fellowship, a fellowship from the Learning Sciences Research Institute at the University of Nottingham, and by IERI grant number REC-043779 to “Learning-Oriented Dialogue in Cognitive Tutors: Towards a Scalable Solution to Performance Orientation”. We would like to thank Angela Wagner, Jay Raspat, Meghan Naim, Katy Getman, Pat Battaglia, Dina Crimone, Russ Hall, and Sue Cameron for assisting in the collection of the data discussed here. We would also like to thank Darren Gergle, Vincent Aleven, Dave Andre, Joseph Beck, and the anonymous reviewers for helpful discussions and suggestions.",
year = "2008",
month = aug,
doi = "10.1007/s11257-007-9045-6",
language = "אנגלית",
volume = "18",
pages = "287--314",
journal = "User Modeling and User-Adapted Interaction",
issn = "0924-1868",
publisher = "Springer Netherlands",
number = "3",

}

Why students engage in "gaming the system" behavior in interactive learning environments

Baker R, Walonoski J, Heffernan N, Roll I, Corbett A, Koedinger K. Why students engage in "gaming the system" behavior in interactive learning environments. Journal of Interactive Learning Research. 2008;19(2):185-224.
 

In recent years there has been increasing interest in the phenomena of "gaming the system," where a learner attempts to succeed in an educational environment by exploiting properties of the system's help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this article, we present three studies, conducted with two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system. We also present a fourth study to determine how teachers' perspectives on gaming behavior are similar to, and different from, researchers' perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive learning environment.

@article{68df616af1bc4b75bd50cf5d530a20d4,
title = "Why students engage in {"}gaming the system{"} behavior in interactive learning environments",
abstract = "In recent years there has been increasing interest in the phenomena of {"}gaming the system,{"} where a learner attempts to succeed in an educational environment by exploiting properties of the system's help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this article, we present three studies, conducted with two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system. We also present a fourth study to determine how teachers' perspectives on gaming behavior are similar to, and different from, researchers' perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive learning environment.",
author = "Ryan Baker and Jason Walonoski and Neil Heffernan and Ido Roll and Albert Corbett and Kenneth Koedinger",
year = "2008",
language = "אנגלית",
volume = "19",
pages = "185--224",
journal = "Journal of Interactive Learning Research",
issn = "1093-023X",
publisher = "Association for the Advancement of Computing in Education",
number = "2",

}

2007

Designing for metacognition-applying cognitive tutor principles to the tutoring of help seeking

Roll I, Aleven V, McLaren BM, Koedinger KR. Designing for metacognition-applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning. 2007 Dec;2(2-3):125-140. https://doi.org/10.1007/s11409-007-9010-0
 

Intelligent Tutoring Systems have been shown to be very effective in supporting learning in domains such as mathematics, physics, computer programming, etc. However, they are yet to achieve similar success in tutoring metacognition. While an increasing number of educational technology systems support productive metacognitive behavior within the scope of the system, few attempt to teach skills students need to become better future learners. To that end, we offer a set of empirically-based design principles for metacognitive tutoring. Our starting point is a set of design principles put forward by Anderson et al. (Journal of the Learning Sciences, 4:167-207, 1995) regarding Cognitive Tutors, a family of Intelligent Tutoring Systems. We evaluate the relevance of these principles to the tutoring of help-seeking skills, based on our ongoing empirical work with the Help Tutor. This auxiliary tutor agent is designed to help students learn to make effective use of the help facilities offered by a Cognitive Tutor. While most of Anderson's principles are relevant to the tutoring of help seeking, a number of differences emerge as a result of the nature of metacognitive knowledge and of the need to combine metacognitive and domain-level tutoring. We compare our approach to other metacognitive tutoring systems, and, where appropriate, propose new guidelines to promote the discussion regarding the nature and design of metacognitive tutoring within scaffolded problem-solving environments.

@article{111dc04d1f104572a4ddf86bf8442c45,
title = "Designing for metacognition-applying cognitive tutor principles to the tutoring of help seeking",
abstract = "Intelligent Tutoring Systems have been shown to be very effective in supporting learning in domains such as mathematics, physics, computer programming, etc. However, they are yet to achieve similar success in tutoring metacognition. While an increasing number of educational technology systems support productive metacognitive behavior within the scope of the system, few attempt to teach skills students need to become better future learners. To that end, we offer a set of empirically-based design principles for metacognitive tutoring. Our starting point is a set of design principles put forward by Anderson et al. (Journal of the Learning Sciences, 4:167-207, 1995) regarding Cognitive Tutors, a family of Intelligent Tutoring Systems. We evaluate the relevance of these principles to the tutoring of help-seeking skills, based on our ongoing empirical work with the Help Tutor. This auxiliary tutor agent is designed to help students learn to make effective use of the help facilities offered by a Cognitive Tutor. While most of Anderson's principles are relevant to the tutoring of help seeking, a number of differences emerge as a result of the nature of metacognitive knowledge and of the need to combine metacognitive and domain-level tutoring. We compare our approach to other metacognitive tutoring systems, and, where appropriate, propose new guidelines to promote the discussion regarding the nature and design of metacognitive tutoring within scaffolded problem-solving environments.",
keywords = "Cognitive tutors, Help seeking, Instructional design principles, Intelligent tutoring systems, Meta-cognition",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
note = "Funding Information: Acknowledgments We would like to thank Ryan deBaker, Eunjeong Ryu, Jo Bodnar, Ido Jamar, Brett Leber, Jonathan Sewall, Mike Konieczki, Kathy Dickensheets, Grant McKinney, Terri Murphy, Sabine Lynn, Dale Walters, Kris Hobaugh and Christy McGuire for their help carrying out these studies. This research is sponsored by NSF Award IIS-0308200, the Graduate Training Grant awarded to Carnegie Mellon University by the Department of Education (# R305B040063), and NSF Award SBE-0354420 to the Pittsburgh Sciences of Learning Center. The contents of the paper are solely the responsibility of the authors and do not necessarily represent the official views of the NSF.",
year = "2007",
month = dec,
doi = "10.1007/s11409-007-9010-0",
language = "אנגלית",
volume = "2",
pages = "125--140",
journal = "Metacognition and Learning",
issn = "1556-1623",
publisher = "Springer New York",
number = "2-3",

}

Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring

Roll I, Aleven V, McLaren BM, Koedinger KR. Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. In Proceedings of the 13th International Conference on Artificial Intelligence in Education AIED 2007. 2007. p. 203-210. (Proceedings of the 13th International Conference on Artificial Intelligence in Education AIED 2007).
 
In our on-going endeavor to teach students better help-seeking skills we designed a three-pronged Help-Seeking Support Environment that includes (a) classroom instruction (b) a Self-Assessment Tutor, to help students evaluate their own need for help, and (c) an updated version of the Help Tutor, which provides feedback with respect to students help-seeking behavior, as they solve problems with the help of an ITS. In doing so, we attempt to offer a comprehensive helpseeking suite to support the knowledge, skills, and dispositions students need in order to become more effective help seekers. In a classroom evaluation, we found that the Help-Seeking Support Environment was successful in improving students declarative help-seeking knowledge, but did not improve students learning at the domain level or their help-seeking behavior in a paper-and-pencil environment. We raise a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within ITSs.
@inbook{29a7e96f1ab343d6890713a2227a21ee,
title = "Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring",
abstract = "In our on-going endeavor to teach students better help-seeking skills we designed a three-pronged Help-Seeking Support Environment that includes (a) classroom instruction (b) a Self-Assessment Tutor, to help students evaluate their own need for help, and (c) an updated version of the Help Tutor, which provides feedback with respect to students help-seeking behavior, as they solve problems with the help of an ITS. In doing so, we attempt to offer a comprehensive helpseeking suite to support the knowledge, skills, and dispositions students need in order to become more effective help seekers. In a classroom evaluation, we found that the Help-Seeking Support Environment was successful in improving students declarative help-seeking knowledge, but did not improve students learning at the domain level or their help-seeking behavior in a paper-and-pencil environment. We raise a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within ITSs.",
keywords = "cognitive tutors, empirical study, help seeking, intelligent tutoring systems, learning, metacognition, self assessment, self regulated",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M} and Koedinger, {Kenneth R}",
year = "2007",
language = "American English",
isbn = "9781586037642",
series = "Proceedings of the 13th International Conference on Artificial Intelligence in Education AIED 2007",
pages = "203--210",
booktitle = "Proceedings of the 13th International Conference on Artificial Intelligence in Education AIED 2007",

}

Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring

Roll I, Aleven V, McLaren BM, Koedinger KR. Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. In Luckin R, Koedinger KR, Greer J, editors, Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work. IOS Press BV. 2007. p. 203-210. (Frontiers in Artificial Intelligence and Applications).
 

In our on-going endeavor to teach students better help-seeking skills we designed a three-pronged Help-Seeking Support Environment that includes (a) classroom instruction (b) a Self-Assessment Tutor, to help students evaluate their own need for help, and (c) an updated version of the Help Tutor, which provides feedback with respect to students’ help-seeking behavior, as they solve problems with the help of an ITS. In doing so, we attempt to offer a comprehensive help-seeking suite to support the knowledge, skills, and dispositions students need in order to become more effective help seekers. In a classroom evaluation, we found that the Help-Seeking Support Environment was successful in improving students’ declarative help-seeking knowledge, but did not improve students’ learning at the domain level or their help-seeking behavior in a paper-and-pencil environment. We raise a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within ITSs.

@inproceedings{682825742ef14796b6f62a6e9f0843ea,
title = "Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring",
abstract = "In our on-going endeavor to teach students better help-seeking skills we designed a three-pronged Help-Seeking Support Environment that includes (a) classroom instruction (b) a Self-Assessment Tutor, to help students evaluate their own need for help, and (c) an updated version of the Help Tutor, which provides feedback with respect to students{\textquoteright} help-seeking behavior, as they solve problems with the help of an ITS. In doing so, we attempt to offer a comprehensive help-seeking suite to support the knowledge, skills, and dispositions students need in order to become more effective help seekers. In a classroom evaluation, we found that the Help-Seeking Support Environment was successful in improving students{\textquoteright} declarative help-seeking knowledge, but did not improve students{\textquoteright} learning at the domain level or their help-seeking behavior in a paper-and-pencil environment. We raise a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within ITSs.",
keywords = "Cognitive Tutors, Empirical Study, Help Seeking, Intelligent Tutoring Systems, Metacognition, Self-Assessment, Self-Regulated Learning",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
note = "Publisher Copyright: {\textcopyright} 2007 The authors and IOS Press. All rights reserved.; 13th International Conference on Artificial Intelligence in Education, AIED 2007 ; Conference date: 09-07-2007 Through 13-07-2007",
year = "2007",
language = "אנגלית",
series = "Frontiers in Artificial Intelligence and Applications",
publisher = "IOS Press BV",
pages = "203--210",
editor = "Rosemary Luckin and Koedinger, {Kenneth R.} and Jim Greer",
booktitle = "Artificial Intelligence in Education",

}

2006

Towards teaching metacognition: Supporting spontaneous self-assessment

Roll I, Ryu E, Sewall J, Leber B, McLaren BM, Aleven V et al. Towards teaching metacognition: Supporting spontaneous self-assessment. In Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings. Springer Verlag. 2006. p. 738-740. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11774303_85
 

The Self-Assessment Tutor (SAT) is an add-on component to Cognitive Tutors that supports self-assessment in four steps: prediction, attempt, reflection, and projection. The SAT encourages students to self-assess their ability spontaneously while problem solving, and to use help resources accordingly. For that reason its episodes precede the students' work with the Cognitive Tutor, which itself remains unchanged. The SAT offers detailed feedback and help function to support the Self-Assessment process. A complementary instruction is given to students before working with the SAT. We hypothesize that working with the SAT will encourage students to self-assess on subsequent problems requiring similar skills, and thus will promote learning. A classroom evaluation of SAT is currently in progress.

@inproceedings{9ef8b2fe7df0458692b0836495c2b917,
title = "Towards teaching metacognition: Supporting spontaneous self-assessment",
abstract = "The Self-Assessment Tutor (SAT) is an add-on component to Cognitive Tutors that supports self-assessment in four steps: prediction, attempt, reflection, and projection. The SAT encourages students to self-assess their ability spontaneously while problem solving, and to use help resources accordingly. For that reason its episodes precede the students' work with the Cognitive Tutor, which itself remains unchanged. The SAT offers detailed feedback and help function to support the Self-Assessment process. A complementary instruction is given to students before working with the SAT. We hypothesize that working with the SAT will encourage students to self-assess on subsequent problems requiring similar skills, and thus will promote learning. A classroom evaluation of SAT is currently in progress.",
author = "Ido Roll and Eunjeong Ryu and Jonathan Sewall and Brett Leber and McLaren, {Bruce M.} and Vincent Aleven and Koedinger, {Kenneth R.}",
year = "2006",
doi = "10.1007/11774303_85",
language = "אנגלית",
isbn = "3540351590",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "738--740",
booktitle = "Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings",
note = "8th International Conference on Intelligent Tutoring Systems, ITS 2006 ; Conference date: 26-06-2006 Through 30-06-2006",

}

Toward Meta-cognitive tutoring: A model of help seeking with a cognitive tutor

Aleven V, McLaren B, Roll I, Koedinger K. Toward Meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education. 2006;16(2):101-128.
 

The research reported in this paper focuses on the hypothesis that an intelligent tutoring system that provides guidance with respect to students’ meta-cognitive abilities can help them to become better learners. Our strategy is to extend a Cognitive Tutor (Anderson, Corbett, Koedinger, & Pelletier, 1995) so that it not only helps students acquire domain-specific skills, but also develop better general help-seeking strategies. In developing the Help Tutor, we used the same Cognitive Tutor technology at the meta-cognitive level that has been proven to be very effective at the cognitive level. A key challenge is to develop a model of how students should use a Cognitive Tutor’s help facilities. We created a preliminary model, implemented by 57 production rules that capture both effective and ineffective help-seeking behavior. As a first test of the model’s efficacy, we used it off-line to evaluate students’ help-seeking behavior in an existing data set of student-tutor interactions. We then refined the model based on the results of this analysis. Finally, we conducted a pilot study with the Help Tutor involving four students. During one session, we saw a statistically significant reduction in students’ metacognitive error rate, as determined by the Help Tutor’s model. These preliminary results inspire confidence as we gear up for a larger-scale controlled experiment to evaluate whether tutoring on help seeking has a positive effect on students’ learning outcomes.

@article{600610fd1731423d80b8fdff10509159,
title = "Toward Meta-cognitive tutoring: A model of help seeking with a cognitive tutor",
abstract = "The research reported in this paper focuses on the hypothesis that an intelligent tutoring system that provides guidance with respect to students{\textquoteright} meta-cognitive abilities can help them to become better learners. Our strategy is to extend a Cognitive Tutor (Anderson, Corbett, Koedinger, & Pelletier, 1995) so that it not only helps students acquire domain-specific skills, but also develop better general help-seeking strategies. In developing the Help Tutor, we used the same Cognitive Tutor technology at the meta-cognitive level that has been proven to be very effective at the cognitive level. A key challenge is to develop a model of how students should use a Cognitive Tutor{\textquoteright}s help facilities. We created a preliminary model, implemented by 57 production rules that capture both effective and ineffective help-seeking behavior. As a first test of the model{\textquoteright}s efficacy, we used it off-line to evaluate students{\textquoteright} help-seeking behavior in an existing data set of student-tutor interactions. We then refined the model based on the results of this analysis. Finally, we conducted a pilot study with the Help Tutor involving four students. During one session, we saw a statistically significant reduction in students{\textquoteright} metacognitive error rate, as determined by the Help Tutor{\textquoteright}s model. These preliminary results inspire confidence as we gear up for a larger-scale controlled experiment to evaluate whether tutoring on help seeking has a positive effect on students{\textquoteright} learning outcomes.",
keywords = "Cognitive modeling, Educational log file analysis, Help seeking, Meta-cognition, Tutor agents",
author = "Vincent Aleven and Bruce McLaren and Ido Roll and Kenneth Koedinger",
year = "2006",
language = "אנגלית",
volume = "16",
pages = "101--128",
journal = "International Journal of Artificial Intelligence in Education",
issn = "1560-4292",
publisher = "Springer US",
number = "2",

}

The help tutor: Does metacognitive feedback improve students' help-seeking actions, skills and learning?

Roll I, Aleven V, McLaren BM, Ryu E, Baker RSJD, Koedinger KR. The help tutor: Does metacognitive feedback improve students' help-seeking actions, skills and learning? In Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings. Springer Verlag. 2006. p. 360-369. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11774303_36
 

Students often use available help facilities in an unproductive fashion. To improve students' help-seeking behavior we built the Help Tutor - a domain-independent agent that can be added as an adjunct to Cognitive Tutors. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive help-seeking model and giving students appropriate feedback. In a classroom evaluation the Help Tutor captured help-seeking errors that were associated with poorer learning and with poorer declarative and procedural knowledge of help seeking. Also, students performed less help-seeking errors while working with the Help Tutor. However, we did not find evidence that they learned the intended help-seeking skills, or learned the domain knowledge better. A new version of the tutor that includes a self-assessment component and explicit help-seeking instruction, complementary to the metacognitive feedback, is now being evaluated.

@inproceedings{ea730194093b40f4a003cabecf4cc03e,
title = "The help tutor: Does metacognitive feedback improve students' help-seeking actions, skills and learning?",
abstract = "Students often use available help facilities in an unproductive fashion. To improve students' help-seeking behavior we built the Help Tutor - a domain-independent agent that can be added as an adjunct to Cognitive Tutors. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive help-seeking model and giving students appropriate feedback. In a classroom evaluation the Help Tutor captured help-seeking errors that were associated with poorer learning and with poorer declarative and procedural knowledge of help seeking. Also, students performed less help-seeking errors while working with the Help Tutor. However, we did not find evidence that they learned the intended help-seeking skills, or learned the domain knowledge better. A new version of the tutor that includes a self-assessment component and explicit help-seeking instruction, complementary to the metacognitive feedback, is now being evaluated.",
author = "Ido Roll and Vincent Aleven and McLaren, {Bruce M.} and Eunjeong Ryu and Baker, {Ryan S.J.D.} and Koedinger, {Kenneth R.}",
year = "2006",
doi = "10.1007/11774303_36",
language = "אנגלית",
isbn = "3540351590",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "360--369",
booktitle = "Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings",
note = "8th International Conference on Intelligent Tutoring Systems, ITS 2006 ; Conference date: 26-06-2006 Through 30-06-2006",

}

Generalizing detection of gaming the system across a tutoring curriculum

Baker RSJD, Corbett AT, Koedinger KR, Roll I. Generalizing detection of gaming the system across a tutoring curriculum. In Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings. Springer Verlag. 2006. p. 402-411. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11774303_40
 

In recent years, a number of systems have been developed to detect differences in how students choose to use intelligent tutoring systems, and the attitudes and goals which underlie these decisions. These systems, when trained using data from human observations and questionnaires, can detect specific behaviors and attitudes with high accuracy. However, such data is time-consuming to collect, especially across an entire tutor curriculum. Therefore, to deploy a detector of behaviors or attitudes across an entire tutor curriculum, the detector must be able to transfer to a new tutor lesson without being re-trained using data from that lesson. In this paper, we present evidence that detectors of gaming the system can transfer to new lessons without re-training, and that training detectors with data from multiple lessons improves generalization, beyond just the gains from training with additional data.

@inproceedings{2c51ce0f9a5147a7bd99ffe63b489c19,
title = "Generalizing detection of gaming the system across a tutoring curriculum",
abstract = "In recent years, a number of systems have been developed to detect differences in how students choose to use intelligent tutoring systems, and the attitudes and goals which underlie these decisions. These systems, when trained using data from human observations and questionnaires, can detect specific behaviors and attitudes with high accuracy. However, such data is time-consuming to collect, especially across an entire tutor curriculum. Therefore, to deploy a detector of behaviors or attitudes across an entire tutor curriculum, the detector must be able to transfer to a new tutor lesson without being re-trained using data from that lesson. In this paper, we present evidence that detectors of gaming the system can transfer to new lessons without re-training, and that training detectors with data from multiple lessons improves generalization, beyond just the gains from training with additional data.",
author = "Baker, {Ryan S.J.D.} and Corbett, {Albert T.} and Koedinger, {Kenneth R.} and Ido Roll",
year = "2006",
doi = "10.1007/11774303_40",
language = "אנגלית",
isbn = "3540351590",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "402--411",
booktitle = "Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings",
note = "8th International Conference on Intelligent Tutoring Systems, ITS 2006 ; Conference date: 26-06-2006 Through 30-06-2006",

}

Adapting to when students game an intelligent tutoring system

Baker RSJD, Corbett AT, Koedinger KR, Evenson S, Roll I, Wagner AZ et al. Adapting to when students game an intelligent tutoring system. In Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings. Springer Verlag. 2006. p. 392-401. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11774303_39
 

It has been found in recent years that many students who use intelligent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises focused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Students using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies.

@inproceedings{ff66595697904043b2dbd3645287a247,
title = "Adapting to when students game an intelligent tutoring system",
abstract = "It has been found in recent years that many students who use intelligent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises focused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Students using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies.",
author = "Baker, {Ryan S.J.D.} and Corbett, {Albert T.} and Koedinger, {Kenneth R.} and Shelley Evenson and Ido Roll and Wagner, {Angela Z.} and Meghan Naim and Jay Raspat and Baker, {Daniel J.} and Beck, {Joseph E.}",
year = "2006",
doi = "10.1007/11774303_39",
language = "אנגלית",
isbn = "3540351590",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "392--401",
booktitle = "Intelligent Tutoring Systems - 8th International Conference, ITS 2006, Proceedings",
note = "8th International Conference on Intelligent Tutoring Systems, ITS 2006 ; Conference date: 26-06-2006 Through 30-06-2006",

}

2005

Detecting when students game the system, across tutor subjects and classroom cohorts

Baker RS, Corbett AT, Koedinger KR, Roll I. Detecting when students game the system, across tutor subjects and classroom cohorts. In User Modeling 2005 - 10th International Conference, UM 2005, Proceedings. Springer Verlag. 2005. p. 220-224. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11527886_28
 

Building a generalizable detector of student behavior within intelligent tutoring systems presents two challenges: transferring between different cohorts of students (who may develop idiosyncratic strategies of use), and transferring between different tutor lessons (which may have considerable variation in their interfaces, making cognitively equivalent behaviors appear quite different within log files). In this paper, we present a machine-learned detector which identifies students who are "gaming the system", attempting to complete problems with minimal cognitive effort, and determine that the detector transfers successfully across student cohorts but less successfully across tutor lessons.

@inproceedings{e8f6ad70dca64f408e63511374705983,
title = "Detecting when students game the system, across tutor subjects and classroom cohorts",
abstract = "Building a generalizable detector of student behavior within intelligent tutoring systems presents two challenges: transferring between different cohorts of students (who may develop idiosyncratic strategies of use), and transferring between different tutor lessons (which may have considerable variation in their interfaces, making cognitively equivalent behaviors appear quite different within log files). In this paper, we present a machine-learned detector which identifies students who are {"}gaming the system{"}, attempting to complete problems with minimal cognitive effort, and determine that the detector transfers successfully across student cohorts but less successfully across tutor lessons.",
author = "Baker, {Ryan Shaun} and Corbett, {Albert T.} and Koedinger, {Kenneth R.} and Ido Roll",
year = "2005",
doi = "10.1007/11527886_28",
language = "אנגלית",
isbn = "3540278850",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "220--224",
booktitle = "User Modeling 2005 - 10th International Conference, UM 2005, Proceedings",
note = "10th International Conference on User Modeling 2005, UM 2005 ; Conference date: 24-07-2005 Through 29-07-2005",

}

Do performance goals lead students to game the system?

Baker RS, Roll I, Corbett AT, Koedinger KR. Do performance goals lead students to game the system? In Artificial Intelligence in Education (AIED). 2005. p. 57-64
@inproceedings{21ca0c81d4d24df59a058bff7205ecb5,
title = "Do performance goals lead students to game the system?",
author = "Baker, {Ryan Shaun} and Ido Roll and Corbett, {Albert T.} and Koedinger, {Kenneth R.}",
year = "2005",
language = "American English",
pages = "57--64",
booktitle = "Artificial Intelligence in Education (AIED)",

}

Modeling students' metacognitive errors in two intelligent tutoring systems

Roll I, Baker RS, Aleven V, McLaren BM, Koedinger KR. Modeling students' metacognitive errors in two intelligent tutoring systems. In User Modeling 2005 - 10th International Conference, UM 2005, Proceedings. Springer Verlag. 2005. p. 367-376. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11527886_48
 

Intelligent tutoring systems help students acquire cognitive skills by tracing students' knowledge and providing relevant feedback. However, feed-back that focuses only on the cognitive level might not be optimal - errors are often the result of inappropriate metacognitive decisions. We have developed two models which detect aspects of student faulty metacognitive behavior: A prescriptive rational model aimed at improving help-seeking behavior, and a descriptive machine-learned model aimed at eliminating attempts to "game" the tutor. In a comparison between the two models we found that while both successfully identify gaming behavior, one is better at characterizing the types of problems students game in, and the other captures a larger variety of faulty behaviors. An analysis of students' actions in two different tutors suggests that the help-seeking model is domain independent, and that students' behavior is fairly consistent across classrooms, age groups, domains, and task elements.

@inproceedings{4efcf8ccbdc14432958b26e01df36358,
title = "Modeling students' metacognitive errors in two intelligent tutoring systems",
abstract = "Intelligent tutoring systems help students acquire cognitive skills by tracing students' knowledge and providing relevant feedback. However, feed-back that focuses only on the cognitive level might not be optimal - errors are often the result of inappropriate metacognitive decisions. We have developed two models which detect aspects of student faulty metacognitive behavior: A prescriptive rational model aimed at improving help-seeking behavior, and a descriptive machine-learned model aimed at eliminating attempts to {"}game{"} the tutor. In a comparison between the two models we found that while both successfully identify gaming behavior, one is better at characterizing the types of problems students game in, and the other captures a larger variety of faulty behaviors. An analysis of students' actions in two different tutors suggests that the help-seeking model is domain independent, and that students' behavior is fairly consistent across classrooms, age groups, domains, and task elements.",
author = "Ido Roll and Baker, {Ryan S.} and Vincent Aleven and McLaren, {Bruce M.} and Koedinger, {Kenneth R.}",
year = "2005",
doi = "10.1007/11527886_48",
language = "אנגלית",
isbn = "3540278850",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "367--376",
booktitle = "User Modeling 2005 - 10th International Conference, UM 2005, Proceedings",
note = "10th International Conference on User Modeling 2005, UM 2005 ; Conference date: 24-07-2005 Through 29-07-2005",

}

An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor

Aleven V, Roll I, McLaren B, Ryu EJ, Koedinger K. An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. In ARTIFICIAL INTELLIGENCE IN EDUCATION. Vol. 125. 2005. p. 17-24. (Frontiers in Artificial Intelligence and Applications).
@inbook{d1afeca1caf8485aa10fdfb4a4d47fa4,
title = "An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor",
author = "Vincent Aleven and Ido Roll and Bruce McLaren and Ryu, {Eun Jeong} and Kenneth Koedinger",
year = "2005",
language = "אנגלית",
isbn = "978-1-58603-530-3",
volume = "125",
series = "Frontiers in Artificial Intelligence and Applications",
pages = "17--24",
booktitle = "ARTIFICIAL INTELLIGENCE IN EDUCATION",

}

An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor

Aleven V, Roll I, McLaren B, Ryu EJ, Koedinger K. An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. In Looi CK, McCalla G, Bredeweg B, Breuker J, editors, Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology. IOS Press BV. 2005. p. 17-24. (Frontiers in Artificial Intelligence and Applications).
 

Given the important role that meta-cognitive processes play in learning, intelligent tutoring systems should not only provide domain-specific assistance, but should also aim to help students in acquiring meta-cognitive skills. As a step toward this goal, we have constructed a Help Tutor, aimed at improving students' help-seeking skill. The Help Tutor is based on a cognitive model of students' desired help-seeking processes, as they work with a Cognitive Tutor (Aleven et al., 2004). To provide meta-cognitive tutoring in conjunction with cognitive tutoring, we designed an architecture in which the Help Tutor and a Cognitive Tutor function as independent agents, to facilitate re-use of the Help Tutor. Pilot tests with four students showed that students improved their help-seeking behavior significantly while working with the Help Tutor. The improvement could not be attributed to their becoming more familiar with the domain-specific skills being taught by the tutor. Although students reported afterwards that they welcomed feedback on their help-seeking behavior, they seemed less fond of it when actually advised to act differently while working. We discuss our plans for an experiment to evaluate the impact of the Help Tutor on students' help-seeking behavior and learning, including future learning, after their work with the Help Tutor.

@inproceedings{78e7e11a086845a691b40dba811d3368,
title = "An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor",
abstract = "Given the important role that meta-cognitive processes play in learning, intelligent tutoring systems should not only provide domain-specific assistance, but should also aim to help students in acquiring meta-cognitive skills. As a step toward this goal, we have constructed a Help Tutor, aimed at improving students' help-seeking skill. The Help Tutor is based on a cognitive model of students' desired help-seeking processes, as they work with a Cognitive Tutor (Aleven et al., 2004). To provide meta-cognitive tutoring in conjunction with cognitive tutoring, we designed an architecture in which the Help Tutor and a Cognitive Tutor function as independent agents, to facilitate re-use of the Help Tutor. Pilot tests with four students showed that students improved their help-seeking behavior significantly while working with the Help Tutor. The improvement could not be attributed to their becoming more familiar with the domain-specific skills being taught by the tutor. Although students reported afterwards that they welcomed feedback on their help-seeking behavior, they seemed less fond of it when actually advised to act differently while working. We discuss our plans for an experiment to evaluate the impact of the Help Tutor on students' help-seeking behavior and learning, including future learning, after their work with the Help Tutor.",
author = "Vincent Aleven and Ido Roll and Bruce McLaren and Ryu, {Eun Jeong} and Kenneth Koedinger",
note = "Publisher Copyright: {\textcopyright} 2005 The authors. All rights reserved.; 12th International Conference on Artificial Intelligence in Education, AIED 2005 ; Conference date: 18-07-2005 Through 22-07-2005",
year = "2005",
language = "אנגלית",
series = "Frontiers in Artificial Intelligence and Applications",
publisher = "IOS Press BV",
pages = "17--24",
editor = "Chee-Kit Looi and Gord McCalla and Bert Bredeweg and Joost Breuker",
booktitle = "Artificial Intelligence in Education",

}

2004

A Metacognitive ACT-R Model of Students' Learning Strategies in Intelligent Tutoring Systems

Roll I, Baker RS, Aleven V, Koedinger KR. A Metacognitive ACT-R Model of Students' Learning Strategies in Intelligent Tutoring Systems. In Lester JC, Vicari RM, Paraguacu F, editors, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer Verlag. 2004. p. 854-856. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-540-30139-4_98
 

Research has shown that students' problem-solving actions vary in type and duration. Among other causes, this behavior is a result of strategies that are driven by different goals. We describe a first version of a computational cognitive model that explains the origin of these strategics and identifies the tendencies of students towards different learning goals. Our model takes into account (i) interpersonal differences, (ii) an estimation of the student's knowledge level, and (iii) current feedback from the tutor, in order to predict the next action of the student -a solution, a guess or a help request. Our longterm goal is to use identification of the students' strategies and their efficiency in order to better understand the learning process and to improve the metacognitive learning skills of the students.

@inbook{88567d7857d9447e93d5aa52492d1211,
title = "A Metacognitive ACT-R Model of Students' Learning Strategies in Intelligent Tutoring Systems",
abstract = "Research has shown that students' problem-solving actions vary in type and duration. Among other causes, this behavior is a result of strategies that are driven by different goals. We describe a first version of a computational cognitive model that explains the origin of these strategics and identifies the tendencies of students towards different learning goals. Our model takes into account (i) interpersonal differences, (ii) an estimation of the student's knowledge level, and (iii) current feedback from the tutor, in order to predict the next action of the student -a solution, a guess or a help request. Our longterm goal is to use identification of the students' strategies and their efficiency in order to better understand the learning process and to improve the metacognitive learning skills of the students.",
author = "Ido Roll and Baker, {Ryan Shaun} and Vincent Aleven and Koedinger, {Kenneth R.}",
year = "2004",
doi = "10.1007/978-3-540-30139-4_98",
language = "אנגלית",
isbn = "3540229485",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "854--856",
editor = "Lester, {James C.} and R.M. Vicari and F. Paraguacu",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

Promoting Effective Help-Seeking Behavior through Declarative Instruction

Ido Roll VA, Koedinger K. Promoting Effective Help-Seeking Behavior through Declarative Instruction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2004;3220:857-859.
 

Research has shown that students' help-seeking behavior is far from being ideal. In trying to make it more efficient, 27 students using the Geometry Cognitive Tutor regularly received individual online instructions. The instruction to the HELP group, aimed to improve their help-seeking behavior, included a walk-through metacognitive example. The CONTROL group received "placebo instruction" with a similar walk-through but without the help-seeking content. In two subsequent weeks, the HELP group used the system's hints more frequently than the CONTROL group. However, we didn't observe a significant difference in the learning outcomes. These results suggest that appropriate instruction can improve help-seeking behavior in ITS usage. Further evaluation should be performed in order to design better instruction and improve learning.

@article{f2b18e6182724f0586423d6f32af1672,
title = "Promoting Effective Help-Seeking Behavior through Declarative Instruction",
abstract = "Research has shown that students' help-seeking behavior is far from being ideal. In trying to make it more efficient, 27 students using the Geometry Cognitive Tutor regularly received individual online instructions. The instruction to the HELP group, aimed to improve their help-seeking behavior, included a walk-through metacognitive example. The CONTROL group received {"}placebo instruction{"} with a similar walk-through but without the help-seeking content. In two subsequent weeks, the HELP group used the system's hints more frequently than the CONTROL group. However, we didn't observe a significant difference in the learning outcomes. These results suggest that appropriate instruction can improve help-seeking behavior in ITS usage. Further evaluation should be performed in order to design better instruction and improve learning.",
author = "{Ido Roll}, {Vincent Aleven} and Kenneth Koedinger",
year = "2004",
language = "אנגלית",
volume = "3220",
pages = "857--859",
journal = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
issn = "0302-9743",
publisher = "Springer Verlag",

}

Toward Tutoring Help Seeking Applying Cognitive Modeling to Meta-cognitive Skills

Aleven V, McLaren B, Roll I, Koedinger K. Toward Tutoring Help Seeking Applying Cognitive Modeling to Meta-cognitive Skills. In Lester JC, Vicari RM, Paraguacu F, editors, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer Verlag. 2004. p. 227-239. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-540-30139-4_22
 

The goal of our research is to investigate whether a Cognitive Tutor can be made more effective by extending it to help students acquire help-seeking skills. .We present a preliminary model of help-seeking behavior that will provide the basis for a Help-Seeking Tutor Agent. The model, implemented by 57 production rules, captures both productive and unproductive help-seeking behavior. As a first test of the model's efficacy, we used it off-line to evaluate students' help-seeking behavior in an existing data set of student-tutor interactions, We found that 72% of all student actions represented unproductive help-seeking behavior. Consistent with some of our earlier work (Aleven & Koedinger, 2000) we found a proliferation of hint abuse (e.g., using hints to find answers rather than trying to understand). We also found that students frequently avoided using help when it was likely to be of benefit and often acted in a quick, possibly undeliberate manner. Students' help-seeking behavior accounted for as much variance in their learning gains as their performance at the cognitive level (i.e., the errors that they made with the tutor). These findings indicate that the help-seeking model needs to be adjusted, but they also underscore the importance of the educational need that the Help-Seeking Tutor Agent aims to address.

@inbook{fea45c72650c4bf1b9aa0991d926a40f,
title = "Toward Tutoring Help Seeking Applying Cognitive Modeling to Meta-cognitive Skills",
abstract = "The goal of our research is to investigate whether a Cognitive Tutor can be made more effective by extending it to help students acquire help-seeking skills. .We present a preliminary model of help-seeking behavior that will provide the basis for a Help-Seeking Tutor Agent. The model, implemented by 57 production rules, captures both productive and unproductive help-seeking behavior. As a first test of the model's efficacy, we used it off-line to evaluate students' help-seeking behavior in an existing data set of student-tutor interactions, We found that 72% of all student actions represented unproductive help-seeking behavior. Consistent with some of our earlier work (Aleven & Koedinger, 2000) we found a proliferation of hint abuse (e.g., using hints to find answers rather than trying to understand). We also found that students frequently avoided using help when it was likely to be of benefit and often acted in a quick, possibly undeliberate manner. Students' help-seeking behavior accounted for as much variance in their learning gains as their performance at the cognitive level (i.e., the errors that they made with the tutor). These findings indicate that the help-seeking model needs to be adjusted, but they also underscore the importance of the educational need that the Help-Seeking Tutor Agent aims to address.",
author = "Vincent Aleven and Bruce McLaren and Ido Roll and Kenneth Koedinger",
year = "2004",
doi = "10.1007/978-3-540-30139-4_22",
language = "אנגלית",
isbn = "3540229485",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "227--239",
editor = "Lester, {James C.} and R.M. Vicari and F. Paraguacu",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

What goals do students have when choosing the actions they perform?

Roll I, Baker RS, Aleven V, Koedinger KR. What goals do students have when choosing the actions they perform? In International Conference on Cognitive Modeling (ICCM). 2004. p. 380-381
@inproceedings{9ab5f488ecbe4a9992cf761d0ac1c4d5,
title = "What goals do students have when choosing the actions they perform?",
author = "Ido Roll and Baker, {Ryan Shaun} and Vincent Aleven and Koedinger, {Kenneth R.}",
year = "2004",
language = "American English",
pages = "380--381",
booktitle = "International Conference on Cognitive Modeling (ICCM)",

}

© 2024 Copyright Elyachar Central Library, Technion - Israel Institute of Technology