Structural Alignment in Comparison

I agree with mrivera that it’s a bit hard to pull generalizations that we can apply to HCI out of this paper without seeing more of how this study fits into the broader space of research on how people think about comparisons/relationships. The author’s findings, however, that people notice more differences between two objects they find similar (objects they can draw a relationship between) is pretty obvious. In HCI practice, this may manifest in trying to make UI components consistent, for example, so that the user can more easily focus on and recall many small differences than being disorientated by large changes.

I worry, however, about trying to draw generalization for HCI out of studies this reductive. This paper studies a narrow, specific perceptual phenomena. Trying to pull too much applicable theme from it, I worry, will dangerously lean towards making specious, non-scientific, and completely opinion-based claims. In HCI we would like to be informed by psychology and cognitive science. How do we do this without completely contorting, misrepresenting or misunderstanding less-applied science?

As for Question 5, I think using words as a test set was one of the more valid, measurable ways for the authors to test their hypothesis. A word carries some depth of meaning, but also an agreed upon dictionary definition. Of course there may be variability of one person’s personal or cultural associations with a word different from another person. However words have pretty standard core meaning. If we open this up to testing people’s comparisons of objects or experiences (I’m sure this must have been studied somewhere) we open up to a much broader range of person biases, associations, that just might be too strong confounds to make any claims on the data.

When encountering a new thing, our go-to mode for interpreting it is to say that "it's like [x] but a little different." It's hard to even imagine how we would process something that was completely different than anything we'd ever encountered before (if we could even conceptualize such a thing). I don't think that this paper was trying to imply that similar things are more different than different things, but rather that it's easier to pick out and verbalize simple distinctions when they're obvious. A red circle is different from an otherwise-identical blue circle because one is red and the other is blue. It takes more effort to think of discrete differences between a red circle and a walrus, even though there are obviously more.

I'm not sure I can think of broad implications for HCI, but I have personal experience with how this is useful in creativity outside HCI in a task somewhat similar to the "draw people who don't exist" task discussed in this paper. In a creative writing class I once took, half of the class was tasked with simply generating ideas for stories while the other half was tasked with generating ideas for stories that started with two people walking out of a building. Despite the latter category being hugely more restricted, those people had a far easier time generating ideas because of the restriction.

I'll take a stab at how this might be applied to HCI: If you're trying to create a new interface, there might be value in departing farther from existing interfaces than making incremental changes. If the differences are only incremental, people may be fixate on the differences since they more readily come to mind. Perhaps a more radical departure (or the addition of additional superficial differences) might weaken the mental comparison with existing interfaces and lead people to consider the new interface more on it's own terms. Obviously, I'm just conjecturing here and there are other aspects which will influence interface acceptance/adoption, but there might be something worth exploring here.

I think the experimental design is fine. I did wonder about the validity of the types of differences. I don't think it really impacts the main findings here, but I'm curious how consistent the labels (alignable vs. non-alignable, types) are. Are those really different categories or are those the categories *these* researchers have focused on. Citing previous work you did to justify the method isn't necessarily the most convincing thing (though that cited paper may well provide strong justification for the technique).

I think the experiment design was clever, and the authors did a commendable job of teasing out a potential confounding variable in Experiment 1 by conducting Experiment 2. 100%-perfect experiments are hard to design. I will stand by them and defend their rigor against critics.

In thinking about how this applies to HCI, it seems like a lot of people are fixated on UI evaluation, but what about implications for other areas of HCI? For example, if it's difficult for people to make comparisons between totally dissimilar items, then maybe one implication is that shopping/recommender systems should provide people with suggested lists of important features to consider instead of making people go through the cognitive struggle of finding nonalignable differences themselves. Along the same lines, maybe inspiration systems for creative tasks should provide people with lists of suggested criteria, constraints, or features to focus on when coming up with analogies to help them think outside the box.

On a kind of random note: One thing that really struck a nerve in me was the last paragraph before the Conclusion about how people put more weight on alignable test scores when making decisions. THIS IS WHAT'S WRONG WITH THE EDUCATION SYSTEM RIGHT NOW!!! Standardized tests are used to evaluate students now, because they're an easy way to produce alignable information on student progress. But there are so many other aspects of learning that can't be as easily quantified or aligned- like grit, discipline, character building, emotional maturity, empathy development, etc. Just because something is easier to align doesn't mean it should be the primary measure for evaluating a person. This really ruffled my feathers. How can we overcome these cognitive biases from structural alignment to make this world a better place?

Building off Felicia's post, I have a (tenuous) example of how this might apply to HCI research that is not about UIs: A lot of the work in Geoff's lab is around encountering, processing, and challenging how we view interpersonal differences (hm, I think this is an accurate statement?). So from this perspective, if we want to prime study participants to notice or mentally engage with differences among people more, this work suggests that we should present participants with humans that lean towards the similar side. And conversely, if we want to participants not to be as readily aware of interpersonal differences, then we should present them with humans that are more dissimilar (?).

This just doesn't seem right, though. Perhaps if users were asked to create a list of differences between such humans, they would generate more ideas for the more similar pair than for the more dissimilar pair, per the findings of this work. But does that doesn't tell us much about how they actually view those differences-- perhaps having a shorter, more salient list of differences still means a lot more (depending on what you are studying, etc.). For example, let's say I want to create an in-group identity among a group of users in an artificial online community for the purposes of a study. I would think that the more similar these users are on a number of different measures, the more readily they will be able to view others in the community as part of their in-group. Even if they can list more differences between themselves and other users in this situation, I would think the connection and extent to which they feel similar to the other participants would still be stronger than if the other participants had been more dissimilar.

(Btw, I don't think this work would refute such an idea. Because obviously people might be able to list more differences between a dog and a cat than between a dog and the Grand Canyon, but then if you ask them which is more similar, they'll still say the dog and the cat, not the dog and the Grand Canyon. But just trying to think of ways to extend...).

I agree with Judith on the argument that similarity making it easier to establish a common ground. The AI example is also very appropriate. I would also argue that more valuable difference can be found when we compare between similar works. For example in HCI research, we always refer previous work and compare our work with them to point out the novelty and importance of our work. There won't be much to dig out if we compare between different research fields.