Post by anhong on Apr 23, 2016 22:37:14 GMT
[written jointly by Toby and Anhong]
Authors
Anand Kulkarni: Was a Ph.D. Student at UC Berkeley Department of Industrial Engineering and Operations Research at the time of this paper. Now president and co-founder of LeadGenius: a startup using human computation and machine intelligence to automate sales at scale.
Matthew Can: Stanford University Computer Science Department. No bio found.
Björn Hartmann: Björn Hartmann is an Associate Professor of EECS at UC Berkeley. His research in Human-Computer Interaction focuses on novel design, prototyping, and implementation tools for the era of post-personal computing. His group investigates how better software and hardware can facilitate the exploration of interactive devices that leverage novel form factors and technologies (e.g., sensors and actuators). He also investigates how software can help students, designers, and makers to learn and share their expertise online. He got his Ph.D. in Computer Science at Stanford with Scott Klemmer.
Venue:
CSCW ‘12
Same year, same conference from the same group:
Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1013-1022. DOI=http://dx.doi.org/10.1145/2145204.2145355
Summary:
Background: Crowdsourcing problems are often solved as a series of microtasks. For quality assurance and to accomplish complex work, multiple microtasks are frequently chained together into workflows. Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
Problem: However, workflow design remains a major challenge. Requesters commonly rely on an iterative process to construct good workflows. The high cost & complexity of workflow design limit participation in crowdsourcing marketplaces to experts willing to invest substantial time. It also limits the kinds of work that are crowdsourced.
Solution: “Turkomatic” system: a novel crowdsourcing tool that allows the crowd to collaboratively design and execute workflows in conjunction with a requester. It accepts a requester’s specification of a broad objective, then asks crowd workers to determine how to structure workflows to achieve the objective.
Turkomatic executes a continuous price-divide-solve loop that asks workers to recursively divide complex steps into simpler ones until they are at an appropriately simple level, then to solve them. Other workers are asked to verify the solutions and combine to results into a coherent answer to the original request. See the below figures for an illustration of 4 kinds of subtasks: subdivision, verification, solution and merging.
Evaluation - Unsupervised Crowd Planning:
The authors test Turkomatic on Amazon Mechanical Turk platform for 4 types of tasks, and have”unsupervised crowd planning” (i.e. crowd be guided algorithmically to plan and solve problems without any input from requesters):
Essay writing: “Write a 3-paragraph essay.”
Natural language query: “Create a list of the names of the Department Chairs of the top 20 computer science college programs in the U.S.”
Itinerary planning: “Plan a complete road trip from San Francisco, California, to New York City. Completely include the location of all necessary hotels, restaurants, and sights along the way.”
Java programming: “Please write a short piece of Java code to reverse a string. The algorithm should take as input a string and output its reverse. Make sure it compiles.”
In the results, we see 2 tasks are successful, which one was snap judgement (turker marks the top-level problem to be solvable, and then the next turker gives a correct solution) and one was successfully solved with complex planning and divided into 7 subtasks.
Two tasks failed because of derailment, which occurred because (1) workers were confused about appropriate task granularity; (2) workers authored subdivisions that could be executed by a single worker, but not split across separate workers and (3) workers who had lost the context of the overall workflow generated decompositions that restated previous tasks as subtasks, leading to cyclic behavior.
One failed because of starvation – after a while, no new workers attempted the available tasks, and the time limit for the experiment expired without the execution of work continuing further.
Evaluation - Collaborative Planning and Execution:
The authors then informally re-ran the experiment for 3 similar tasks for 2 conditions, where condition 1 uses an expert crowd of workers recruited at UC Berkeley; and second uses a Mechanical Turk crowd, augmented through active requester participation and intervening using Turkomatic’s monitoring interface. All task succeeded for both conditions.
Discussion Question:
1. Three tasks used in this paper for evaluating unsupervised crowd planning failed because of starvation or derailment in the recursive workflow generating process. Do you think non-expert turkers can ever reliably design workflow for complex tasks? Why or Why not?
2. The authors examined the effectiveness of unsupervised crowd planning, as well as collaborative planning. Do you think this is enough? Should the authors also compare their system against letting the requesters come up with the workflow on paper, and using the Turkomatic system? How would you do the evaluation?
3. What kind of information do you think is missing from the different steps of the Turkomatic workflow that made it fail in the unsupervised crowd planning part? How can it be improved to better support this “meta-crowdsourcing” of workflows?
4. Collaboration and coordination are important challenges in designing HCI systems, how can we leverage theories in communication, collaboration, etc to inform this area of research?
5. What can we learn about crowdworkers and the design of crowd computing systems from this paper?
Authors
Anand Kulkarni: Was a Ph.D. Student at UC Berkeley Department of Industrial Engineering and Operations Research at the time of this paper. Now president and co-founder of LeadGenius: a startup using human computation and machine intelligence to automate sales at scale.
Matthew Can: Stanford University Computer Science Department. No bio found.
Björn Hartmann: Björn Hartmann is an Associate Professor of EECS at UC Berkeley. His research in Human-Computer Interaction focuses on novel design, prototyping, and implementation tools for the era of post-personal computing. His group investigates how better software and hardware can facilitate the exploration of interactive devices that leverage novel form factors and technologies (e.g., sensors and actuators). He also investigates how software can help students, designers, and makers to learn and share their expertise online. He got his Ph.D. in Computer Science at Stanford with Scott Klemmer.
Venue:
CSCW ‘12
Same year, same conference from the same group:
Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1013-1022. DOI=http://dx.doi.org/10.1145/2145204.2145355
Summary:
Background: Crowdsourcing problems are often solved as a series of microtasks. For quality assurance and to accomplish complex work, multiple microtasks are frequently chained together into workflows. Such workflows may decompose larger tasks into smaller subtasks, and later recompose subtask solutions into an overall work product.
Problem: However, workflow design remains a major challenge. Requesters commonly rely on an iterative process to construct good workflows. The high cost & complexity of workflow design limit participation in crowdsourcing marketplaces to experts willing to invest substantial time. It also limits the kinds of work that are crowdsourced.
Solution: “Turkomatic” system: a novel crowdsourcing tool that allows the crowd to collaboratively design and execute workflows in conjunction with a requester. It accepts a requester’s specification of a broad objective, then asks crowd workers to determine how to structure workflows to achieve the objective.
Turkomatic executes a continuous price-divide-solve loop that asks workers to recursively divide complex steps into simpler ones until they are at an appropriately simple level, then to solve them. Other workers are asked to verify the solutions and combine to results into a coherent answer to the original request. See the below figures for an illustration of 4 kinds of subtasks: subdivision, verification, solution and merging.
Evaluation - Unsupervised Crowd Planning:
The authors test Turkomatic on Amazon Mechanical Turk platform for 4 types of tasks, and have”unsupervised crowd planning” (i.e. crowd be guided algorithmically to plan and solve problems without any input from requesters):
Essay writing: “Write a 3-paragraph essay.”
Natural language query: “Create a list of the names of the Department Chairs of the top 20 computer science college programs in the U.S.”
Itinerary planning: “Plan a complete road trip from San Francisco, California, to New York City. Completely include the location of all necessary hotels, restaurants, and sights along the way.”
Java programming: “Please write a short piece of Java code to reverse a string. The algorithm should take as input a string and output its reverse. Make sure it compiles.”
In the results, we see 2 tasks are successful, which one was snap judgement (turker marks the top-level problem to be solvable, and then the next turker gives a correct solution) and one was successfully solved with complex planning and divided into 7 subtasks.
Two tasks failed because of derailment, which occurred because (1) workers were confused about appropriate task granularity; (2) workers authored subdivisions that could be executed by a single worker, but not split across separate workers and (3) workers who had lost the context of the overall workflow generated decompositions that restated previous tasks as subtasks, leading to cyclic behavior.
One failed because of starvation – after a while, no new workers attempted the available tasks, and the time limit for the experiment expired without the execution of work continuing further.
Evaluation - Collaborative Planning and Execution:
The authors then informally re-ran the experiment for 3 similar tasks for 2 conditions, where condition 1 uses an expert crowd of workers recruited at UC Berkeley; and second uses a Mechanical Turk crowd, augmented through active requester participation and intervening using Turkomatic’s monitoring interface. All task succeeded for both conditions.
Discussion Question:
1. Three tasks used in this paper for evaluating unsupervised crowd planning failed because of starvation or derailment in the recursive workflow generating process. Do you think non-expert turkers can ever reliably design workflow for complex tasks? Why or Why not?
2. The authors examined the effectiveness of unsupervised crowd planning, as well as collaborative planning. Do you think this is enough? Should the authors also compare their system against letting the requesters come up with the workflow on paper, and using the Turkomatic system? How would you do the evaluation?
3. What kind of information do you think is missing from the different steps of the Turkomatic workflow that made it fail in the unsupervised crowd planning part? How can it be improved to better support this “meta-crowdsourcing” of workflows?
4. Collaboration and coordination are important challenges in designing HCI systems, how can we leverage theories in communication, collaboration, etc to inform this area of research?
5. What can we learn about crowdworkers and the design of crowd computing systems from this paper?
?
