judy
New Member
Posts: 22 
|
Post by judy on Apr 24, 2016 2:54:58 GMT
(Judy & Joselyn) The Wisdom of Crowds
by James SurowieckiThe Author
- Columnist at The New Yorker, The Financial Page
- Ph.D. in American History from Yale
- In 1995, was Editor-in-chief of the financial adivice site The Motley Fool's culture site on America Online, "Rogue."
- More on the book.
- His TED Talk on the wisdom of crowds and the blogosphere.
The Reading
Big ideaIf a group of people make a prediction independently (not collaborating), and their answers are aggregated and averaged, they will predict the outcome with excellent accuracy. Errors essentially cancel each other out, and the right answer emerges. “In other words if you run 10 different jelly bean counting experiments, it’s likely that each time one or two students will out-perform the group. But they will not be the same students each time.” What a group needs to be smart: - Diversity
- Independence
- Decentralization
Examples from the chapterWithin 30 minutes after the Challenger exploded in 1986, the stock market cast “blame” on one of the three manufactures involved in building the Challenger. Thiokol, the o-ring manufacture, stock declined 12%. It took six months, before it was determined the o-rings were indeed the source of the problem. In sports betting, the bigger the group of gamblers, the more accurate the odds are. Essentially the bigger the crowd, the smarter. Because of this, in less popular sports where the crowd of gamblers is smaller and perhaps less diverse, professional gamblers have the opportunity to outsmart the system and make more money. Surowiecki argues that google's search results are built on the wisdom of crowds. Google interprets the link from page A to page B as a vote for page B. If page A is popular, than this vote is weighted as more important. Our reflection
In these examples the crowd is predicting an outcome. These predictions can be proven either right or wrong (i.e. there is an exact number of jellybeans in the jar). Is this wisdom? How about making decisions where there is no clear right or wrong (e.g. choosing the right partner)? Even though the crowd may predict outcomes correctly, it may not always be put to good use. How do you decide how large, diverse, independent and decentralized a crowd needs to be to counter biases, prejudices and misconceptions? Discussion Other readings this week take a more detailed, operationalized and/or critical look at crowds. In this space, we invite you to imagine how, why and if crowdsourced decision-making has a place in our daily lives. 1. Imagine when in your daily life you would want the crowd’s help to make a decision. Describe the problem and then design an intervention using the wisdom of the crowd. Consider who (or what data source) makes up that crowd and what motivates them to weigh in on your decision. 2. Describe your own research or find an example of an existing app or study that relies on crowdsourced decision-making. How does it work? Do you think it meets the requirement for a smart crowd (diverse, independent, decentralized)? What are the benefits and costs? Would you use it? 3. Conversely, imagine a decision in which you would absolutely NOT want the crowd’s input. Why? Imagine a dystopian situation where this future (or current) decision is made through the crowd. What are the consequences? Who is the crowd? In your dystopian vision, what was the justification for harnessing the crowd? 4. This chapter lists examples of when the crowd was right. Can you find an example of when a crowdsourced prediction was wrong? Where did it go wrong? Why? What were the consequences? We encourage you to use illustrations, diagrams and other visual aids to tell your story. |
|
|
|
Post by julian on Apr 24, 2016 16:48:28 GMT
Random thoughts:
- If we are so smart as a crowd how come so many bad choices are taken every day by so many people around the world, a recent one caused by bad group decision making: the 2008 recession.
- According to the reading more than half of the people should be right so .. most people are taking the right choices or more are thinking about the right choice then they meet someone change their mind and well that’s it, they end up taking a wrong choice which kind of makes sense with the other readings.
- The last will correspond to basically violating the independence principle.
- What does this say then about independent thought and heard mentality?
- Crowds are smart but only when the conditions for it are right? How can we know the conditions are right? Namely how can we measure diversity of opinion, independence, decentralization and aggregation?. For instance, say we poll a group of people, how from their responses alone can we make sure they are a wise crowd.
- Besides the above conditions, the crowd wisdom only works when error in information cancels out. This means that all of the individual bias in average cancels out which is similar to assume that the error on the information of each individual follows some symmetric distribution. Now, this is likely not true for small samples.
- However, by the central limit theorem (CLT), any big enough sample, regardless of the original distribution, will be approximately normally distributed. If my interpretation is correct, assuming CLT holds: 1. as long as the sample is big enough there is no problem and the wisdom of Crowds will hold 2. This is a really cool justification on why the wisdom of crowds work.
|
|
|
|
Post by sciutoalex on Apr 24, 2016 20:58:26 GMT
I love thinking of dystopian futures. I think most dystopian crowdsourcing daydreams would come from systems designers creating systems that respect the requirements of independence, decentralization, and diversity but fail to account for the desires of those who are affected by the decisions. An easy set of dystopias is crowds deciding personal choices like whom to love, where to live, or what career to have. In each case, crowds could make these decisions, and objectively they would choose better! Yes marry the high-earning business major and not the drama student. Yes move to Idaho where there is a huge need for IT experts instead of living in Brooklyn. Yes major in nursing because of all the old people! But these decisions would not account for the desires of the individual who will have to live with the decisions. I think these dystopic visions do point to the value of crowdsourcing in personal life. They can offer objective opinions about situations that may *feel* super subjective. But perhaps you should marry the business major. The drama that motivates the drama major really isn't healthy for you or your future...
Another set of dystopias violates the independence, decentralization, and diversity requirements. Imagine if everyone you know got to vote on love, location, or career. You bring a date to Thanksgiving and everyone pulls out their phones and swipes left or right at which point your date gets a text to either stay or leave. Or your coworkers do that for promotions. Here the crowd workers are neither independent, decentralized, or diverse. But they do know you much better. We once lived in a society where there were mechanisms like this. Marriages brokered parent-to-parent were the primary method of dating prior to a few decades ago. We got our careers based on what our parents had, a very rudimentary form of crowdsourcing. Gossip was just a method of intra-worker communication in these tasks.
In these dystopias, the crowd is either so anonymous that it does not care who you are or what your needs are or so close that its decisions are based on the biases of the its members. Why would these systems ever be created? For the massive crowd, the system is more efficient and makes better decisions at the expense of individual preferences. Most Americans now meet each other on dating websites that use algorithms to match people up, so the dystopia is not so far away.
|
|
nhahn
New Member
Posts: 22 
|
Post by nhahn on Apr 25, 2016 18:44:22 GMT
Continuing on Alex's points, if (in some future) we rely on the wisdom, or average of the crowd for all decisions, would there ever be the chance to push back against the status quo. It seems that if we leverage the crowd for judgements, the status quo would just be continuously reinforced, with no opportunity for adjustment. Could a minority ever successfully voice their opinion about an injustice if we always relied on the majority average opinion? Especially in situations where the observable outcome benefits the larger community (on average), but greatly detriments a particular community. It seems like this type of thinking could work well if the outcome distribution was normal, but in situations where you might have a bimodal distribution, this "average" thinking could be detrimental. For example, think of Wikileaks. The opinions regarding the "goodness" of Wikileaks is vastly different -- there are those that view it a necessary liberty while others that view it as an evil unconcealing of secret information. Now, imagine we had a search engine that not only considered popularity, but also reputation of a particular website. If we took the average of the wikileaks reputation, then if very well could get a 3 out of 5 rating. That would be a pretty hard website to find compared, even though a significant portion of the population considers it crucial information.
|
|
|
|
Post by judithodili on Apr 25, 2016 22:18:56 GMT
I'm kind of surprised that groups make "better decisions" - There were a bunch of papers that we read in ARM about group decision making. These papers below were really excellent for understanding the dynamics and a of the research out there about the topic. The consensus is that in group settings, people tend to make decisions along with the crowd, or go with the most knowledgable person, and in extreme situations, try to find a middle ground. So it's interesting that this paper reports that they generally make better decisions - it's understandable if people go along with people who are experts or can game the system, but other than that, not sure how to rationalize their findings.
"Weingart, L. R. (1997). How did they do that? The ways and means of studying group process. Research in Organizational Behavior, 19, 189-239. Pp. 212-228. "
"Tajfel, H., Billig, M. G., Bundy, R. P., & Flament, C. (1971). Social categorization and intergroup behavior. European Journal of Social Psychology, 1, 149-178. "
"Kim, H. and Markus, H. R. (1999). Deviance or uniqueness, harmony or conformity? A cultural analysis. Journal of Personality and Social Psychology, 77, 785-800."
"Frank, M. G., & Gilovich, T. (1988). The dark side of self- and social perception: Black uniforms and aggression in professional sports. Journal of Personality and Social Psychology, 54, 74-85"
that discusses group decision making,
|
|
aato
New Member
Posts: 16
|
Post by aato on Apr 25, 2016 23:26:26 GMT
I have so many thoughts about crowds and dating. I'm going to try to be concise. Lauren McCarthy is a really awesome computer scientist / artist. She creates technologies as design probes / art projects. She did a project called "Social Turkers: Crowdsourced Dating" where she set up a web cam during a first date and had the crowd give her feedback on what to do and how to respond. She did some rounds of iteration to scaffold the feedback (which she received via text) and would get comments like "act more excited about what he just said" or "appear more interested" or "tell him a secret." This work for sure freaks me out, but also was kind of cool, like having a friend around to keep an eye on you and if you are socially awkward - help you survive the trials of online dating. lauren-mccarthy.com/socialturkers.com/My second thought is about the Bachelor / Bachelorette - a dating show where a single man or woman starts out with 20-30 potential partners and each week kicks a few people off the show until the last week when usually a marriage proposal happens. America is totally obsessed with this show and the social media element for the past few years has been in full force reacting after and during every episode. Most of the relationships on that show do not work out. I partially attribute this to bad decision making, partially to the concept that you could find someone to marry in isolation in a few months from a small and pre-selected pool. However, I wonder if the viewers at home are better at predicting a good match right off the bat in this 'wisdom of the crowds' style. There's a lot of bias issues here at play as well - viewers at home only see what the producers put into the final cut of an episode - an episode they put together after already knowing the final outcome. But I wonder if you had a live audience hanging out or live streaming during that first night if they would be good at determining the best long-term partner. |
|
|
|
Post by jseering on Apr 26, 2016 0:04:17 GMT
If you want to see an example of bad crowd cognition, just try asking reddit for dating advice. Not only will advice vary tremendously across different subreddits (and the same person might give a different response to the same post in a different subreddit), but initial responses will shape the discussion. I wonder if reddit dating advice would be better if opinions were aggregated from individuals without their seeing each other's opinions.
These sorts of examples work when opinions of the crowd are distributed relatively normally around the "ideal," but they don't work when there's an inbuilt bias where the average person is inclined to have a shifted opinion (more or less by definition). The internets are full of "you won't believe this one simple fact!" and quite often people are surprised by that one simple fact.
|
|
mkery
New Member
Posts: 22
|
Post by mkery on Apr 26, 2016 0:29:01 GMT
jseering dating advice on reddit is pretty amusing for that reason. Can we differentiate situations in which we want the statistical aggregate opinion of the impersonal Crowd (amazon reviews for a cat litter box), versus the opinion of a smaller group of individuals (dating advice perhaps)? Thinking of sciutoalex ’s dystopia, I’m reminded of a science fiction story in which people walk around with all of their family members in their head after those family members die (and thus lose their own bodies). This is a very different kind of small, personal crowd, weighing-in on your every choice. Like judithodili talks about group dynamics, small groups behavior in more centralized ways and have social hierarchy on decisions. A small crowd of 15 family members mentally on access at all times may boil down to your mother pushing the strongest advice to you. A small crowd also feels personal, and possibly way more like a violation of privacy. Would you rather broadcast personal decisions to a small community or giant anonymous crowd?
|
|
toby
New Member
Posts: 21
|
Post by toby on Apr 26, 2016 1:46:27 GMT
Another example of bad crowd cognition would be Twitch Plays Pokemon, or basically any "Twitch Plays" games (https://www.twitch.tv/directory/game/Twitch%20Plays). For those of you who are not familiar with the game, it's an online streaming game, where everyone watching (the crowd) can send instruction for the game character using the chat box. It used to be that every single instruction will get , but then they change that to a voting for the next move every 30 seconds. As one may imagine, the game almost had zero progression under "the wisdom of the crowd". There has been attempts to collaborate and coordinate but all ended up with failure.
So I guess what we learned from this experiment is that, it's hard to have the crowd coordinate with each other.
|
|
|
|
Post by mmadaio on Apr 26, 2016 2:12:25 GMT
Toby, I thought Twitch Plays Pokemon actually got fairly far in the game in the anarchy mode, and just got further, and did it more efficiently, when they moved to the voting system.
More generally, this chapter seems to overlook many of the obvious criticisms of group decision-making that are elucidated more clearly in the other readings this week. The naivete in this "crowd wisdom" seems akin to the (disingenuous?) optimism of anarcho-capitalist, libertarian, free trade enthusiasts, who believe that, once government intervention is removed, the wisdom of the market will lead to ____ (prosperity, jobs, wealth, widespread happiness, etc etc). This might have worked in theory, or in the wishful thinking and abstract models of 18th century economists, but in reality, we see that the "invisible hand" of the market is actually the hand of the people who stand to benefit most from government de-regulation. It's a nice fantasy to believe that the wisdom of the market (crowd) will self-correct any of the negative outcomes, but when small decisions made at high levels of the system lead to cascading series of decisions at the individual level that inevitably lead to benefits for those at the top, that wishful thinking about crowd/market wisdom is untenable.
|
|
|
|
Post by xiangchen on Apr 26, 2016 2:38:11 GMT
As I read the chapter, I kept thinking maybe the crowd's wisdom could be just statistics. For example, consider guessing the jelly beans. we can hypothesize that the expected error governed by an underlying distribution is zero - imagine asking all 6 billion people on this planet to make a guess and compute the average. If this is the case, the crowd giving a closer guess is because a larger sample size better represents the underlying distribution by counterbalancing variances, thus resulting in being closer to the expected error (zero). Thinking in terms of limit, a crowd of N people beat n individuals less and less as n continually approximates N.
This also begs a question - when would the crowd fail to serve superiorly? One possible scenario is there is systematic 'shift' of people's ability to judge or accomplish a task, making the crowd unable to add up to a better solution. For example, consider the chess board illusion example. Will the crowd split the choice in halves and sum up to the two squares being the same color? Probably not. If the problem is pretty random, like buying a lottery ticket, the crowd also probably couldn't sum up to anything better than an individual chance.
|
|
|
|
Post by fannie on Apr 26, 2016 2:42:34 GMT
I find the examples people bring up about relationships interesting. I was also thinking of the Lauren McCarthy project that Alexandra already mentioned - I think it’s an interesting concept in letting others control how you act so a date goes well, but at the same time it could take away from you being you, or potentially how unique a date is (though perhaps different people, environments, etc. might be enough to make up for it). Also there’s a difference between your friends (or just people you know) giving you advice vs random people like on reddit. It’s tricky when it comes to other people because everyone has different expectations, values, goals, and viewpoints, whereas for something like gambling everyone just wants to make more money. But that kind of personal information would be known by your friends. If any random person, on the other hand, was given those parameters, I’m still skeptical that they’d be able to make the best decision… and as was mentioned before, I suppose online dating sites are already doing this by collecting your information and preferences. Pushing it further - having the crowd help you make decisions throughout your relationships could also be super weird, there’d be that whole ‘dating the crowd’ vs ‘dating you’ thing.
Also for Twitch Plays Pokemon - they still beat the game(s) in some number of days I think.
|
|
|
|
Post by cgleason on Apr 26, 2016 2:47:09 GMT
I'm very fond of the idea of wisdom of crowds, but it somewhat assumes that we can take an outside view of the problem at hand, where any crowd member is capable of obtaining and processing information just as well as any other member. In the jelly bean scenario, this is true. Everyone can see the same jar. In the challenger example, this is true. they checked for insider trading. In sports and politics, this is true. Hell, I trust the political prediction markets more than any pollster. But it's worth noting the limitations here: the jelly bean example is largely inconsequential, the stock traders would never have pointed specifically at the o-ring as the culprit, and prediction markets are still only as good as the news they are reading. Making more nuanced claims about what was to blame in the Challenger explosion or creating new political theories requires expert knowledge.
Now this expert knowledge could be an in the head of a single expert such as Feynman. In this case, we have to find a way to weight this expert opinion higher than other crowd members. Alternatively, it could be the "emergent behavior" of a humanity-wide collective superintelligence (see Nick Bostrom's book Superintelligence). This touches on all of the points made by previous commentators: crowd opinions are great, but learning a means of coordination is amazing. Does this mean that decentralization and individuality can't work in a coordinated crowd? No, because I think without those you just end up with a biased crowd. It's largely going to rely on creating a decentralized system that is also capable of coordination (see Twitch Plays, Wikipedia (maybe?), Bernstein's crowd research).
|
|
|
|
Post by Anna on Apr 26, 2016 2:54:38 GMT
Ah, first aato thanks for introducing me to Lauren McCarthy, I just checked out some of her projects on her website and find some of what she's done really interesting. This chapter opens with a discussion of Who Wants To Be A Millionaire, where there are clear right/wrongs. And actually, many of the examples in this chapter fall into that category-- who's going to win the NFL playoff, who was primarily responsible for the Challenger disasters, etc. Thinking of dystopian futures, branching off other people's comments: I think it's dangerous to rely on crowdsourcing for any decision where there isn't a clear optimum-- which really, isn't that everything? Maybe it's okay for small choices that don't have clear optimal solutions. For example, choosing what to eat at a restaurant is always stressful for me, and it would be a load off to have the crowd just tell me what to order. (Though even there, I feel wary-- if I can't even make small decisions, how can I learn to make the big ones? What crucial decision making skills am I allowing to atrophy, and how might that affect my life/cognitive function in other ways? Eg could that make me more susceptible to Alzheimer's later in life?). And for bigger decisions, eg sciutoalex 's marriage example: let's say actually that in all respects, we would actually be happier following the crowd than making our own decision. But again, still here I feel wary: what do we miss out on by not feeling uncertain, by not questioning ourselves, by making the wrong decision? What if everyone is just happy all the time? What happens to our creativity, what happens to our cognition, etc? Personally I can get really anxious and feel really off balance if I go too long without feeling wildly upset--which, ok, maybe is an issue I need to deal with... but I think there's still something to be said for a world that is inclusive of a broad spectrum of human experiences, good or bad, that includes conflict and disaster and tragedy and terrible things. (Aka maybe utopia is the true dystopia.) |
|
|
|
Post by stdang on Apr 26, 2016 3:35:46 GMT
3. Conversely, imagine a decision in which you would absolutely NOT want the crowd’s input. Why? Imagine a dystopian situation where this future (or current) decision is made through the crowd. What are the consequences? Who is the crowd? In your dystopian vision, what was the justification for harnessing the crowd?
Imagine a toy that was also your play partner. Like any good sidekick, the toy is faithfully by your side through all your adventures through the mysterious lands of make believe, but there are times in your journey when indecision rears its ugly head. Should I try to rescue the unicorn that is trapped on the top of the playset, making the trecherous journey over the top of the playground playset? Of course this is the perfect opportunity for my play partner to chime in and suggest my next course of action. Intelligent technologies are not quite up to par in their ability to make sense of the environment and make proper judgements about age appropriate play actions. This is where human input would be great. What if we could tap into the variety and wisdom of the crowd to suggest and converge on suggested actions? To some degree, this is Twitch for pretend play, and in the twitch world, agents are able to successful accomplish objectives. However, in the world of young children, the room for error is much smaller and the influence of small negative actions in the grand scheme of all actions encouraged by the system is possibly sufficient for very catastrophic circumstances.
|
|
