Cognitive Tutors - Discussion

Seeing the essentially flat correct-answer curve, I had a similar set of questions that you have Nick. How does this make students feel to progress and become smarter, but have no sense of reward or acknowledgement of their progress? I hope the cognitive tutor does give students a sense of their real improvements, and I assume the authors didn't feel such a implementation detail was worth mentioning. In some ways, constant increasing challenge would increase students' capacities for extended-focus, working through difficulties, etc. These are valuable skills, but these skills can't be reduced to "knowledge components," can they? So it's funny, a highly cognitive model of learning might also be helping do much more teaching than just skill achievement.

I also connected this reading to Bjork's remembering and forgetting paper because it seems that Cognitive Tutors work on the basis of strengthening the information retrieval process. By showing many instances, the student strengths the retrieval mechanism while also abstracting the process from specific instances. I wonder how Cognitive Tutors could use Bjork's insight that forgetting can be useful to learning. Bjork mentions that changing locations is a kind of forgetting, but for most elementary-school learning, it occurs in a single room that only changes once per year. Would using the cognitive tutor in different locations increase student's retention of skills? What if recess were twice as long but the tutors were used outdoors?

Could cognitive tutors be used for non-cognitive learning? Has that been looked at? Can a cognitive tutor teach me to procrastinate less or have more determination to work through difficult problems or situations? Could it teach me to be more comfortable in social situations?

I'm so completely out of the world of intelligent tutoring systems, that I feel somewhat out of sorts, so please bear with me in this post.

Nick raises a lot of the same questions that I had when reading this article. Namely, I am left considering how the responsiveness of the ACT-R system to one's needs as a learner would potentially fall short of one's needs as a human student. I quite relished the periods of time where I felt really strong in an academic area that generally challenged me, like say Calculus, and I feel like those periods where I was among the leads in class on understanding a topic, helped buoy my self-esteem during periods of the class when I felt less capable. While the paper does address some of the personality factors present in students, I still fear that a system like the one described could potentially be the most effective at teaching me new material, yet not fully access my humanity and those social psychological factors therein.

Having just reviewed Bjork's article, I like Alex, was struck with the thought of how some of the principles of his work re: learning, forgetting, and remembering, could be incorporated into the ACT-R model in a way that alters situational factors so that forgetting could be more easily a conduit for remembering. I liked Alex's suggestions about altering the location of where the student interacts with the intelligent tutor. From a design perspective, I'm curious if temporal factors i.e. times of day, avatar and general aesthetic modifications, tasks accomplished or games played, and general demands of the interaction could all be manipulated in an attempt to encourage forgetting and re-learning/salience of information.

There is a ton of Cognitive Tutor research out there on how/when to encourage students, and how research that shows that being using a polite tone in tutors produce better learning effects. It seems like a "duh", but the point is, the way some tutors are designed already take into consideration encouragement, reinforcement, and cognitive load in the way the students are being tutored. I personally have no problems with breaking down content into knowledge components, because I liken cognitive tutors to real tutors (e.g after school supplemental tutors/homework assistants) vs general education administrators. The function of a tutor is to fill in knowledge gaps in a way that encourages the student to master the material and be interested in learning more and Cog. Tutors do that fairly well.

My issue with the ACT-R based and other Cog. Tutors in general is that they are designed one kind of student in mind. There is currently research that shows that the way students interact with tutors are largely dependents on their cultural dispositions towards asking questions, working independently, and feeling comfortable with failure. This leads the tutors to be used in ways other than those they were designed for (which messes up the data the teacher gets for the areas that they need to work on), and also doesnt take into account the cultural diversity of the students that attend American schools. Like everything in HCI - the answer to how these tutors can support students needs is very dependent on a variety of factors and I dont think they do nearly as good a job supporting this diversity. This may make them not only detrimental for some students learning, but may potentially give the teachers false information on the areas they need to work the students to improve their knowledge gaps.

This paper demonstrated a model of organic relationship between basic research, applied research, and field testing. With a theory behind, people can better understand how a system work, why it works and why it doesn’t work. Hypothesis can be generated from theory, and can be tested in testing. However, I feel not all research in HCI are embracing this model. Except for learning sciences and social computing, I think very few theories are being actively used, generated, or tested in technical HCI research. Is this a problem? Why there’s not many besides Fitts's law and GOMS?

About the point of “Maybe even to encourage students to admire how far they've traveled before forging onward? Though it might also be the case that a carefully calibrated challenge reward enough?”, I think there’s a difference between motivation and learning. Motivating the students to keep them learning is certainly important. But when admiring how far they’ve traveled before, that moment students are not learning much. Instead, I agree with the paper’s point that the learning happens most efficiently for strengthening weak skills.

* In one of the comments above, Alex Sciuto said “In some ways, constant increasing challenge would increase students' capacities for extended-focus, working through difficulties, etc. These are valuable skills, but these skills can't be reduced to "knowledge components," can they?”. But in some work on intelligent tutoring systems that support students’ acquisition of metacognitive and self-regulated learning skills, these actually are modeled as (reduced to) “knowledge components” (then again, one might argue – maybe post-hoc – that these were simply intended as ‘useful’ models and needn’t have any psychological reality).

Nick asked: ‘Are there other ways an ACT-R framework falls short of the "real thing" in education?’
ACT-R has always struck me as an extremely flexible framework (not necessarily in a good way). So, as a follow-up question for discussion:
Is it possible for the ACT-R framework itself to ever actually fall short of the “real thing” in education? Or is it always the case that particular models are just “incomplete” and need refinement/extension.


* "...education is most efficient when it focuses students most directly on those individual knowledge components that have relatively low strength..."

In general – focusing only on ‘cognitive’ factors -- I don’t think this claim (at least, the use of the word “most”) is particularly well established. This might imply that an optimal (let’s say, in terms of learning gains per time spent, as measured by some pretest and posttest that are reasonably well-aligned with an intelligent tutoring system’s curriculum) problem selection algorithm would, at any decision step, select a problem targeting a student’s weakest knowledge components (KCs). This has been operationalized in various ways in intelligent tutoring systems – such as “always select the problem with the lowest average estimated probability of mastery, across all within-problem KCs” -- sometimes under the label of ‘cognitive mastery’.

One element that’s absent in many models of student learning that draw upon ACT-R is a model of human ‘forgetting’/decay (though there is much research in the learning sciences supporting spaced practice, and some ITS work that directly responds to this work). I think this may be in part because early intelligent tutoring systems were designed to be used over relatively short time scales, where the effects of forgetting were assumed to be negligible. Under a naïve problem selection algorithm that targets KCs with low estimated mastery, the system will present KCs with high estimated mastery less and less as time progresses, and thus will have fewer and fewer opportunities to re-measure mastery (in a context where humans forget, this will cause the student model to diverge from reality).
Separately, this assumption does not capture the potential importance of more complex/higher-order sequencing effects (e.g. potential benefits of interleaved practice for concept learning, and potential prerequisite structure among KCs… even though there is no “transfer” between KCs, by definition).

I'm interested in the way many of us seem to be trying to fit the role of the tutor in with or in place of the traditional role of the teacher, and we seem to be comparing this sort of simulated problem-based education with the "real thing," which is something more applied (e.g. Fannie's real world examples). This seems like a definitional problem to me, like what we talked about with emotion. If we define emotion as something ephemeral or somehow uniquely human, machines won't ever have it, but if we describe it as something directly procedural then machines already do have it.

In the same vein, if we talk about learning as something semi-spiritual (growing as a person, etc etc) it's much harder to understand whether cognitive tutors are effective. In order to create an effective teaching tool, we really need a better definition of what learning really is in the way that we mean it in the classroom. If learning is answering questions objectively correctly, then cognitive tutors can teach. If learning is growing as a person, we probably need teachers. It's a complicated question, but we should at least try to question our assumptions.

Following up on Joseph's point regarding the frighteningly deep question of "what is learning" I somewhat wonder about the ability of cognitive tutors to successfully true learning. While there is learning for the sake of standardized tests, etc. as mentioned by others, at least I view the benefit of learning as being able to apply learned concepts to unsolved problems in the world. This is somewhat in line with what the Bjork article put forward as "learning". Therefore, to truly learn concepts, can cognitive tutors provide the variety to form both a strong and fully formed abstraction? My fear from cognitive tutors is that they suffer from the same problem of traditional learning, in introducing and recalling concepts in the same environment over and over (like the type of questions asked etc.. ). Do alternative learning environments and instruction (like project based learning) offer benefits on this front? I feel like there might be literature on this, but I don't do learning science research XD.