nhahn
New Member
Posts: 22 
|
Post by nhahn on Mar 27, 2016 3:54:33 GMT
About the authorsNiki — You should hopefully know him by now  . Andrew Peters — Research Programmer working with Niki Abdigani (Abdi) Diriye — Currently working on a startup (used to be at IBM Watson). Has done previous research on information retrieval in an HCI context. Mike Bove — CMU HCI Undergrad Trupti Telang — HCI Masters Student What is sensemakingPeople do a lot of informal learning online through this process of “sensemaking”. Sensemaking is type of bottom up learning, where individuals sample an information space and form a mental modal based on the information encounters. This mental modal is often incorrect or incomplete at the beginning and goes through several revisions. Often times, this model is then used for developing inferences about a topic, commonly through a decision making process. The StudyThe authors wanted to explore what the cost structure for externalizing and generating model looked like. Specifically, they had 3 challenges: How much / what information needs to be captured (to possibly share with others)? What is the cost of adding structure? When should individuals add structure? They created a sensemaking support system that allowed individuals to clips pieces of information and organize the based on <item, dimension, value>. They then manipulated when an individual would assign the dimension and value — either when the clip of information was captured, or during the review near the end of the process. Cool Results
Pushing structuring towards the end resulted in better categories / structures. But it didn’t appear to increase workload or time required. There was no difference in recall for remembering how much individuals remembered about the cameras / car seats between the conditions The mental models in all conditions changed somewhat drastically (even if they said they were already familiar with the domain) QuestionsDo you really thing that there is no additional cost for saving structuring until later? The authors thought there would be, but their hypothesis H1.3 was disconfirmed. Was this just a result of the particular study task not being complex enough? Or is there some other factor in play here? The authors note (somewhat subtly) that their satisficer / maximizer scale was able to account for a large amount of variation for how enjoyable they found Clipper, how helpful they found it, and the amount of time they spent reviewing information. What do you think this says about the usefulness of categorization support systems for individuals? How does this relate back to the desk organization paper? Categorization seems to play many different roles. It can be used for communicating an idea / feature set, for consolidated inference / storage for a single individual (like a decision making task), and for future retrieval, as seen in the desk organization study. Do you think there can be sweeping claims made about categorization, or do they depend on their application area?
|
|
|
|
Post by mrivera on Mar 27, 2016 4:15:09 GMT
I don't buy that there is no additional cost for saving structuring until later. In this study, the task was clear and in a familiar domain (shopping). Had the task been understanding something like String Theory in Physics, I'm not sure the results would turn out the way they did. If a more complex task/topic was provided, I imagine the time would increase exponentially. Here are some questions about the study as a whole: 1) Does sense-making necessitate that an individual actually understands/makes personal sense of the thing versus just clipping bits and grouping them for later? How do we judge whether someone has "made sense" of something? 2) Cost of capturing information - They used time as a measure of difficulty for the task, I'm not sure if this was the best description of difficulty. Perhaps perceived difficulty is a better measure. Time doesn't carry that much weight  Nit pick: There were a lot of hypotheses floating around- were these mainly used to build up a more compelling story? Couldn't this paper have been accomplished without listing all the things. |
|
mkery
New Member
Posts: 22 
|
Post by mkery on Mar 27, 2016 13:54:29 GMT
I was curious why there wasn’t a condition for no tool at all. A deeper question struck me by the paper introduction: yes, tools may be insufficient for internet sensemaking, but how necessary are tools at all? In a typical search task, we hold and refine a mental model of the camera we are searching for, or the research topic on google scholar. Does this need to be externalized?
On one hand, we find it very important to document long-term searches such as our google scholar literature search on a given project. I save recipes on cooking blogs to try out later. However on short-term “research” questions, such as which cat litter box or which camera to buy, is creating external documentation overkill? I can see both sides of this argument, I only wish the paper had explored no-tool further in some condition.
|
|
Qian
New Member
Posts: 20
|
Post by Qian on Mar 27, 2016 16:11:09 GMT
Michael R. raised an interesting point about other sense-making tasks such as String Theory. Obviously the study of this paper only deals with dimension-structured tasks, and implicitly assumes (1) the foraging process is a bottom-up one: seeking unstructured information to build a systematic understanding; (2) sense-making is rational. However, there are structured information online (i.e. wiki pages, amazon filters) that are meant to save sense-making efforts, and at least part of the structure-building efforts seem can be automated. It can be interesting to further look into how people’s perception of dimension importance in the process of mental model building.
Regarding sense-making rationality, I think the tool described in this paper, and other sense-making supports in general, make people think more reflectively upon the decisions they make. That’s why I agree with Mary Bath that the claim would be stronger with a control condition (no tool used). I don't buy that there is no additional cost for saving structuring until later because without the tool maybe people won’t come back to structuring at all. But the additional time cost shifts the user from fast thinking to a slow thinking channel.
|
|
|
|
Post by Anna on Mar 27, 2016 18:09:33 GMT
As one of the co-discussion leaders, I don't have a ton to add to Nathan's post because he did such a great job summarizing, etc (great work, buddy). But some more discussion questions to consider:
Do we need mental models?
-(Continuing with Mary Beth and Qian's discussion about how necessary are tools)-- To take it a step further, how often do we encounter decisions/tasks (in online contexts) in which we don't actually want to make sense? What about when we want to just go by a feeling/our gut? Or alternately, are there cases where we want to appear or trick ourselves into believing that we're making a rational decision/engaging in sensemaking, but actually, we still are going by feeling/gut?
More complicated application areas (building off Nathan's question, Mike's point about complexity of tasks):
-What do you all do you in your own research when you're doing a literature review of a new topic area? Assuming you all use some sort of tool or organizational system as you're foraging and sensemaking: at what point and to what extent do you impose structure and organization? If you've played around with different methods, what are the affordances and tradeoffs of different ways you've experimented with?
|
|
|
|
Post by jseering on Mar 27, 2016 20:51:20 GMT
It certainly makes sense to me that committing to a set structure too early in the process is problematic. The set of folders I initially created to house research papers when I started here was essentially useless three months later because the categories were no longer meaningful (even though the papers themselves were). We see this in a related sense in design under the label "fixation." However, saving structuring until very late in the process can still have negative effects, and like mrivera I'm skeptical of the finding in this paper that there was no cost.
Per Anna's question, I think there's a lot of potential for exploration of what we do when we don't explicitly create categories. What are our subconscious or "gut" modes of structuring things? I'm not sure I'd know how to study this, as there are certainly problems with asking people to articulate what they're doing subconsciously, but I think often our first step toward creating structures happens before we're aware of it. I wonder if there's a study to be done perhaps with eye-tracking software that might capture our subconscious associations between things before we're consciously aware of how we're associating them.
|
|
|
|
Post by judithodili on Mar 28, 2016 16:16:18 GMT
Completely agree with mrivera... I suspect these results are due to the simplicity of the task, and suspect that they'll be different in a unknown domain. I also agree with Joseph that committing to a structure early on can be problematic because the better the mental model someone forms with a topic, the more likely they are to re-organize their thoughts into a structure that is more meaningful to them. This would be impossible to do earlier on due to the inadequacy of their mental model. This goes back to the other paper (the desktop organization one) that structure has to be very flexible and encouraging of change to be effective for users.
|
|
|
|
Post by sciutoalex on Mar 28, 2016 20:10:40 GMT
I really like Anna's point about situations where we don't want mental models. In those situations where we want to trick ourselves. I like that observation because I would bet the majority of the time we shop online, we're actually in this state. We want to *feel* we have enough of a mental model to convince us that the thing we're about to buy is the right one. I'd be curious to see systems that instead of trying to help you find the right product instead try to help you understand that all the products you are looking at are the same and nothing matters. But I think that's besides the point of this paper...
Back in the world of research, I take issue with the String Theory counter-point because learning String Theory has nothing to do with categorization. There is a difference between learning about something and being able to categorize it. I think if you showed a person a variety of Wikipedia articles on competing theories of physics, you could elicit similar results that the authors found with photo equipment. The categories would likely be superficial—key people, general acceptance by the scientific community, important implications of the theory—but they'd be categories nonetheless. An expert looking at the same articles would probably come up with 1) many more categories that are 2) much more detailed. The interplay between expertise and categorization is interesting, but I don't think it negates the authors' insight that we can separate foraging from sense making in useful ways.
I would have loved to see a follow up study that intertwines foraging and sense making. Perhaps the user skims the articles for ten key points, then they categorize those points, then they go back and find facts that fill in the nascent schema. I would think that interweaving the two tasks would give better results. I would think it would be especially powerful if an interface could be made that shows side-by-side the rubric and the text.
|
|
|
|
Post by xuwang on Mar 28, 2016 20:33:00 GMT
For question 1, I also think the type of task may play a role here, but in a different direction from previous posts. I think for more complicated tasks, to structure the information later would save more time.
In the case of purchasing cameras, if we already have a specific dimension in mind, for example we're looking for cameras that are smaller in size, or have a good appearance, or at least we're familiar with what functions cameras have, I think condition 1 will cost less time, as people can put in information quickly as they read the pages. However, if cameras are completely new to us, what I usually do in searching for items is opening several pages at the same time, read them back and forth, and then structure the information. If i have to come up with some dimensions at an early stage, that'd be difficult and costs more time. So I think for tasks that we don't have a mature mental model of, structuring the information at a later stage is actually helpful and saves time.
This also reminds me of learning style theory in the learning science literature, where learners are categorized as active vs. reflective learners. active learners will try things out first, and reflective learners will think things through first. learners with low working memory capacity tend to prefer an active style of learning, whereas learners with high working memory capacity tend to prefer a reflective style of learning. Similarly, I think people who have low working memory tend to prefer condition 1 where they can note things down immediately, and people who have high working memory tend to prefer condition 2&3, after reviewing all pages and then structure the information.
I think the interface proposed in this paper is one approach to address the information categorization problem brought up in the organizing desk paper. In that paper, we were discussing whether it's good to label/classify files into folders immediately or save them for later, and the timing issue is discussed in this paper. Also i think assigning pages with multiple dimensions instead of giving them only one label, and combining the dimensions later is helpful for the future retrieval of the pages.
|
|
|
|
Post by Felicia Ng on Mar 28, 2016 21:39:50 GMT
This type of sensemaking support system seems to be useful for tasks in which there is a very specific goal (i.e. make a decision on which camera or carseat to buy), but I feel like a lot of information foraging and sensemaking in our lives is done without an explicit or immediate goal in mind. As we bumble through life, we seek and take in a lot of information from both the external world and our internal states to form this complex and messy understanding of what things are and how things work, and then later on, when a question or decision comes up, we go back through our memory and sift out the relevant pieces of subconsciously consolidated knowledge to form a coherent answer to the inquiry. I guess what I'm pointing out is that I see two types of sensemaking: short-term/explicit/immediate and long-term/subconscious/delayed. The support system described in this paper only addresses the former. I wonder if there are any computerized tools that could help with the latter. If so, what are they? And if not, why not? What are the challenges or limiting factors?
|
|
|
|
Post by Yang Zhang on Mar 29, 2016 0:02:23 GMT
Based on Mary Beth, Qian and Anna's discussion on whether there should be a mental model for sensing making. I agree that the mental model is very volatile which makes every tool such like the one proposed in this paper difficult to support. It remembers me of the synthesis tool we tried to build last semester where our conclusion is that a piece of white paper and a pen is the best tool to support a variety of mental models. However, as said in the first class, building a mental model help us frame the unknown and is a useful tool to tackle problems.
I'm also a bit skeptical about the no-additional-cost argument in the paper, based on everyday practice.
|
|
toby
New Member
Posts: 21
|
Post by toby on Mar 29, 2016 1:00:17 GMT
I also think intuitively pushing structuring towards the end would incur additional cost base on my experience, and hope there can be follow up study using different set of tasks on different domains to see if the conclusion made by the authors is universally true. Related to my own research, as I find smartphone users seldom type into structured documents like spreadsheet for various reasons, I think this kind of system can be particularly useful on mobile devices. It also not only allows easier information retrieval and sense making for the user, but also enables better machine readability on the data.
|
|
|
|
Post by nickpdiana on Mar 29, 2016 1:38:34 GMT
Joseph, those "subconscious associations" might not be completely inaccessible to researchers. Though imperfect, we can use brain imaging to look at representations of conceptual models, and more importantly, how those models change with learning. A Post-Doc in my old lab has done this with physics concepts ( paper here), comparing physics novices to experts, and modeling the changes in neural representation over the course of a semester-long course. I'd be interested to see how those changes map to changes in the user-generated "software model." I'd be willing to suspect that our brains generate a much different (and maybe more efficient) model, and that the software necessarily adds some noise. It might be interesting though to see where the software excels over our natural mental model. |
|
|
|
Post by francesx on Mar 29, 2016 1:46:49 GMT
My first reaction to this paper and the results was "WE NEED A TOOL LIKE THIS FOR LEARNING!!" Creating a mental model for complex domains, as Mike says, certainly can be time consuming. Or not, depending on your previous knowledge. For example, this study [1] explored how beginners and experts in the domain of physics classify problems. The study looked at elements within the problem definition/solution space that each group used to classify the problems. My understanding of the paper is that experts have a mental model very different compared to that of novices. Years of practice and work certainly can lead to that, but a tool as the one described in the paper sounds very compelling for education.
An addition to the above point: in learning sciences we talk a lot about learners models. A tool that would be able to help the learner create and maintain a mental model could be an indispensable addition to the learner model (and could potentially foster good self-regulated learning behavior!!)
Chi, Michelene TH, Paul J. Feltovich, and Robert Glaser. "Categorization and representation of physics problems by experts and novices*." Cognitive science 5.2 (1981): 121-152.
|
|
|
|
Post by xiangchen on Mar 29, 2016 1:53:58 GMT
I like how this paper is able to abstract and reduce information structuring to the <item, dimension, value> model. However, it did not strike me as 'categorization'; rather I was thinking more about 'labeling'. And I am also curious about whether this format of foraging information is applicable to any area. Intuitively it looks like it simulates closely how people gather and organize information, where the goal is to arrive at a level of view beyond individual entries of the sought data. What about tasks where the mental model is not the sum of the sought information but a mapping of it. For example, planning for a trip might involves simply 'summing up' points of interest documented by previous travelers; it might also involves 'mapping' such information to the making of a unique itinerary that, in an extreme case, ends up being nothing like any of those previous routes. Perhaps this stage is too evolved and cannot be afforded by the a simple 'labeling' model.
|
|
. 
