Chan, Joel, Steven Dang, and Steven P. Dow. “Improving Crowd Innovation with Expert Facilitation.”

Discussion Leader: Shiwani

Summary:

As the title suggests, this paper studies whether crowd innovation can be improved through expert facilitation. The authors created a new system which builds on the strategies used during face-to-face brainstorming and uses this to provide high-level “inspirations” to crowd-workers to improve their ideation process.

The first study compared the creativity levels of ideas generated by a “guided” crowd with ideas generated without any facilitation. The study showed that the ideators in the condition with expert facilitators generated more ideas, generated more creative ideas and exhibited more divergent thinking. The second study focused on the abilities of the facilitator and involved novice facilitators, keeping all other constraints same. Surprisingly, the facilitation seemed to negatively influence the ideation process.

Reflections:

This paper touches on and build on many things we have been talking about this semester. One of the key ideas behind <SystemName> is feedback, and its role in improving creativity.

I really liked the paper as a whole. Their approach of adapting expert facilitation strategies from face-to-face brainstorming to a crowd-sourcing application was quite novel and interesting. They took special efforts to make the feedback synchronous in order to “guide” the ideation, as with real-time brainstorming.

They make a strong case for the need for something such as <SystemName>. Their first point centers around the fact that the crowd-workers may be hampered because of inadequate feedback (as we have discussed before in the “feedback” papers). And the second point is that the existing systems were not built to scale. With <SystemName> the authors created a system to provide expert feedback while also ensuring it could scale by keeping the feedback at a higher level, rather than individualized.

The authors mention that a good system requires divergent thinking as well as convergence of ideas. The divergence prevents local minima in the ideation, and the convergence allows for growth of promising solutions into better ideas. This was an interesting way of looking at the creative process. And this situates their choice of  using a skilled facilitator as a tool.

The study was quite well-designed with clear use-cases. On one-hand they wished to study the effect of having a facilitator guide the ideation. And a second study captured the effect of the skill-level of the facilitator. The interface design was simplistic, both for the ideators and the facilitators. I liked the word-cloud idea for the facilitators- it is a neat way to present an overview/insight at such a scale. I also liked the “pull” model for inspiration, where the ideators were empowered to ask for inspiration whenever they felt the need for it as opposed to pre-determined check points. This deviates somewhat from the traditional brainstorming where experts choose when to intervene, but again, for the scale of the system and the fact that the feedback was not individualized, it makes sense.

 The authors do mention that their chosen use-case may limit the generalization of their findings, but the situational, social case was a good choice for an explorative study.

As with a previous paper we read, creativity was qualified by the authors, due to its subjective nature. Using novelty and value as evaluative aspects of creativity seems like a good approach, and I liked that the creativity score was a multiplication of these two to reflect their interactive effect.

Questions:

1. A previous paper defined creativity in terms of novelty and practical (being of use to the user, and it being practically feasible to manufacture in today’s age), whereas this paper focused only on the “of value” aspect in addition to novelty. Do you think either definition is better than the other?
2. The paper brings forth the interesting notion that “just being watched” is not sufficient to improve worker output. Do you think this is specific to creativity, and the nature of the feedback the workers received?
3. For the purpose of scale, the authors gave “inspirations” as opposed to individualized feedback (like Shepherd did). Do you think the more granular, personalized feedback would be helpful in addition to this? In fact, would you consider the inspirations as “feedback”?

Read More

Chan et al., “Improving Crowd Innovation with Expert Facilitation” CSCW’16

Discussion Leader (con): Nai-Ching

Summary

Although crowdsourcing has been shown to be useful for creativity tasks, the quality of creativity is still an issue. This paper demonstrates that the quality of crowdsourced creativity tasks can be improved by introducing experienced facilitators in real time work setting. The facilitators produce inspirations that are expected to facilitate the ideation. To measure the quality, divergence (fluency and breadth of search), convergence (depth of search) and creative outcomes (rated creativity of ideas) are used. The result from first experiment shows that with the help of experienced facilitators, both the number of generated ideas and max creativity of the output increase. The result of second experiment reveals that with novice/inexperienced facilitators, the creativity of the output is reduced. To further analyze the causes/reasons of the difference, the authors code the strategies that are used to generate the inspirations into categories including “Examples”, “Simulations” and “Inquiries”. While “Examples” and “Inquires” do not have significant effects on the output, “Simulations” are highly associated with higher max creativity of ideas. The authors also point out that the different intentions of experienced and novice facilitators might attribute to the different results of facilitation. The experienced facilitators tend to actually do the facilitating job while the inexperienced facilitators are more inclined to do the ideating job

 

Reflections

It seems to be contradictory that the paper first mentions that popularity and “rich get richer” effects might not be actual innovative potential but later on the facilitation dashboard, the keywords are sized by frequency which seems to be just another form of popularity.

It is not clear about the interaction between ideators and the “inspire me” function before the facilitator enters any inspiration. If there is no inspiration available, is the button disabled? And how do ideators know if there is new inspiration? Also, do facilitators know if ideators request inspiration? I think the “inspire me” function should help keep the workers and lower the attrition rate but based on the results, there is no significant difference between facilitated and unfacilitated conditions.

In addition, the increased creativity only happens at max creativity not including mean creativity. One the one hand, It makes sense as the authors argue that what innovators really care about is increasing the number of exceptional ideas and since it is more likely to get higher creativity with proper facilitation or say proper facilitation increases the potential of getting higher creativity, the proper facilitation is a good technique. On the other hand, it also shows the technique might not be reliable enough to avoid the manual effort of going through all the generated ideas to pick out the good ones (max creativity). This paper also reminds me of an earlier paper we discussed, “Distributed analogical idea generation: inventing with crowds”, which mainly increases the mean creativity and the change of max creativity is not reported. It might be possible to combine both techniques to both increase mean and max creativity of ideas.

It also seems to me that in addition to soliciting more ideas, keeping a good balance between divergence and convergence is also very important but I didn’t see in the future work section that it is important/helpful to show information of current breadth and depth of idea/solution space to the facilitator to help him/her divert the direction of inspirations.

It is interesting to see that one of the themes in ideators’ comments about inspirations provoking new frames of thinking about the problem but actually there is no significant difference of breadth between facilitated and unfacilitated conditions. So I wonder how general the theme is.

Questions

• What reasons do you think cause the discrepancy between user perception and actual measurement of breadth search in the solution space?
• What is the analogy between the technique from this paper and the technique from “Distributed analogical idea generation: inventing with crowds”?
• Can most people appreciate creativity? That is, if a lot of people say something is creative, is it creative? Or if something is creative, do most people think it creative as well?

Read More

Dontcheva, Mira, et al. “Combining crowdsourcing and learning to improve engagement and performance.” Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014.

Discussion Leader (Pro): Sanchit

Summary

This paper presented a crowdsourcing platform called LevelUp For Photoshop. This tool helps workers learn Photoshop skills and tools through a series of tutorials and then allows them to apply these skills to real world image examples from several non-profit organizations that require image touchups before uploading the images for use.

This sort of crowdsourcing platform is different in that it is aimed at completing creative tasks through the crowd but also allowing the crowd to learn a valuable skill that they can apply to other fields and scenarios outside of this crowdsourcing platform. The platform starts off every user with a series of very interactive and step-by-step guiding tutorials. These tutorials are implemented as an extension for Adobe Photoshop which allows the extension to monitor what tools and actions the users have taken. This creates a very easy-to-use and learn-from tutorial system because every action has some sort of feedback associated. The only thing this tool can’t do is judge the quality of the transformations of these images. That task however is extended onto other Amazon MTurk workers who look at a before/after set of images to determine the quality and usefulness of the picture editing job done by a crowd worker in LevelUp.

This paper presented a very thorough and detailed evaluation and study of this project. It involved 3 deployments where each contribution of the approach was added onto the plugin for user testing. The first deployment was only of the interactive tutorial. The authors measured the number of levels the players completed and got helpful feedback and opinions about the tutorial system. The second deployment added the challenge mode and evaluated the results with logs, MTurk worker quality checks and expert quality examination. These photo edits were scored using a point system between 1-3 for usefulness and novelty. The last deployment added real images from non-profit organizations. The test was to determine whether different organizations have a different effect on a user’s editing motivation and skills. The results weren’t as spectacular, but they were still positive in that the skills learned by the users were helpful.

Reflection

Usually crowdsourcing involves menial tasks that have little to no value outside of the platform service, but the authors in this paper designed a very unique and impressive methodology for users to both learn a new and useful skill like photo editing and then applying the skills to complete existing real-world photo editing tasks. They took advantage of the need for certain people to learn Photoshop or image editing and while teaching them were also able to accomplish a real-life photo editing task, thus killing two birds with one stone. Crowdsourcing doesn’t necessarily have to involve monotonous tasks and nor do crowd workers have to be paid monetarily. This is a creative approach where the incentive is the teaching and skills developed for photo editing along with having achievements and badges for completing specific tasks. They may not be as valuable as money, but it is enough incentive to garner interest and leverage the newly learned skills to accomplish an existing task.

The authors conducted extremely detailed surveys and collected feedback from a pool of approximately ten thousand works over a period of 3 years. This type of dedication for evaluation and the associated results of this study prove the usefulness of this type of crowdsourcing platform. It shows that not all crowd work has to be menial tasks and that users can actually learn a new skill and apply the work outside of crowd platforms. However, I do admit that the way these results were presented were non-trivial. The inclusion of graphs, charts or tables would have made it easier to follow along instead of interpreting the numerous percentages within the paragraphs.

By having MTurk workers and experts judge photo edits, they bring in the perspective of an average user and what their perception of quality or usefulness is and they also bring in the perspective of a professional to see how quality or usefulness is judged through their eyes. That, in my opinion, is a pretty strong group of evaluators for such a project especially considering the massive scale at which people volunteered and completed these evaluation tasks on MTurk.

Lastly, I was really impressed by the Photoshop extension that the authors developed. It looked very clean, sleek and easy to learn from because it doesn’t seem to intimidate users like the palette of tools that Photoshop presents. This sleekness can allow workers to retain the skills learned and apply it to future projects that they may have. I think photo editing is a fabulous skill to have for anyone. You can edit your photos to focus or highlight different areas of the pictures or to remove unwanted noise or extraneous subjects from an image. By having a straightforward, step-by-step and interactive tool such as LevelUp, one can really increase their Photoshop skillset by a huge margin.

Questions

• How many of you have edited pictures on Photoshop and are “decent” at it? How many would like to increase your skills and try out a learning tool like this?
• Having a great tutorial is a necessity for such a concept to work where people can both learn and apply those skills elsewhere without their hands being held. What features do you think such tutorials should have to make them successful?

Read More

Dontcheva, Mira, et al. “Combining crowdsourcing and learning to improve engagement and performance.” Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014.

Discussion Leader (con): Ananya

Useful Link:  http://tv.adobe.com/watch/prototype/creative-technologies-lab-photoshop-games/

Summary

This paper discusses how crowdsourcing and learning can be combined to create an environment that benefits both worker and requester. The learning should be such that the skill developed helps workers not only in crowdsourcing context but also marketable in other contexts.

The authors developed a learning interface “LevelUp” on top of Adobe Photoshop. This interface presents an interactive step by step tutorial for photo editing as “missions”, ordered by increasing difficulty level. The tutorial provides sample images for users to work on or users can use their own images. The users are presented with one step at a time and the user has to finish this step to go to the next round. Each mission is associated with points and the number of points increase with the difficulty of the mission. Users can also earn badges on successfully completing a mission which they can share on social networking sites. The system also gives instant feedback to users on their progress. It has 12 tutorials, divided into three levels. At the end of each level, users test their skill in a challenge round. Images in the challenge round are supplied by requester organization.

The interface has two parts. First part is just the interactive tutorial and the challenge round comes in the next part. The challenge round was created to support crowdsourcing. Unlike the interactive part which presents a set of steps for improving an image, the challenge part just suggests improvements and also lets user improvise.

The implementation and results are divided across three deployments. Deployment 1 consisted only of the interactive tutorial. For evaluation, the authors measured number of missions completed by players, collected player’s feedback monthly, interviewed 6 players and finally compared user behavior before and after the game. Overall, this deployment received positive feedback with more than 60% completing atleast level 1.

The deployment 2 tested whether skills learnt in deployment 1 could be used in real world tasks. This included both interactive tutorial and challenge rounds. The authors performed 3 types of evaluations: 1. behavioral logs — number of images edited and types of edit performed, 2. MTurk workers compared original image to edited image and 3. experts examined the quality of edited images and rated between 1-3 on the basis on usefulness and novelty. The results were mixed. However images in challenge 3 received higher “more useful” rating than challenge 1. The authors derives that the users who went till level 3 were more motivated to learn and do a good job.

In deployment 3, the authors added real-images from requesters from 4 different organizations to the challenge round to find out if certain type of institutions would receive better results over others. They deployed this in two different versions — one that included detailed information about the requester organization and the other that just listed the name of the organization. They analyzed behavioral data that included details about the images edited, qualitative assessments from MTurkers, experts and two requesters, and survey data on user-experience. One of the requester assessed 76 images and rated 60% of edited images better than original and the other assessed 470 images and rated 20% of them better than original.

 

Reflection

When I read this paper, the first thing that came to my mind is “Where is the crowdsourcing aspect?”. The only crowdsourcing part was assessment done by MTurkers who needed no specific skill to do the task. Even that part was redundant since the authors were also getting assessment done by experts. I think the title of the paper and the claim of combining crowdsourcing and learning is misleading.

The participants were not crowd workers who were paid to do the job but rather people who wanted  to prettify their images. Now photoshop on its own being a bit overwhelming, LevelUp seemed to be an easy way to learn basics of photo editing. This is an anecdotal view. However this raises the same question that Dr. Luther (sorry Dr. Luther I might not have quoted you accurately) raised yesterday “Would the results differ if we randomly selected people to either play the game or do the task?”. Does the cause of any action influence the results? It would have been interesting ( and more relevant to the title of the paper) to see if MTurkers (who may or may not have interest in photo editing) were chosen as participants and asked to do the learning and take challenges. If they were not paid for the learning part, would they first sit through the learning part(tutorials) because the skills developed might help them somewhere else or they would directly jump to challenges round and take up the challenges because thats where the money is. Even the authors mentioned this point in ‘Limitations’.

The results presented were convoluted — too many correlation made with no clear explanation. I wish they had presented their analysis in some sort of visual format. Keeping track of so many percentages was hard, at least for me.

It is normally  interesting and easy to learn basic editing technics such as adjusting brightness, contrast, saturation, etc.  But to make novices interested in learning advanced technic is a test of the learning platform. The stats provided did not answer the question “what percentage of novices actually learnt advanced technics?” One of the results in deployment 2 says only 74% of users completed challenge 1, 57% challenge 2 and 39% challenge 3 with no explanation on why so less percentage of people continued till challenge 3 and what percentage of novices completed each challenge.

I am also not convinced with their measure of “usefulness”. Any image, even with basic editing, usually looks better than original and as per the definition, it work will get the highest “usefulness” rating. I wish they had a fourth measure in their scale, say 4, which depended on what kind of technics were used. The definition of “novelty” looked flawed too. I mean it works well in this scenario but a crowdsourcing platform like Amazon Mechanical Turk where  workers are used to getting paid for following instructions as nearly as possible may not show much novelty.

With all the issues that are there, still there were a few things I liked. I liked the idea of offering students to practice their skills not through sample use cases but through real scenarios where their effort may benefit someone or something. I also like the LevelUp interface, it is slick. And as I said earlier photoshop may be overwhelming, so an interactive step by step tutorial definitely helps.

Finally, I thought that the skill gained through such tutorials are good only for limited use or, as we have seen in previous discussions, for the task at hand. But without further knowledge or standard recognitions, I doubt how marketable these skills will be outside.

 

Questions

• Do you also think there was no crowdsourcing aspect in the paper apart from a few guidelines mentioned in ‘Future work’?
• Do think the skills developed in similar platforms  can be marketed as advanced skills? How would you change the platform so that the learning here can be used as a professional skill?
• Do you think the results would have been different if the users were not interested participants but rather MTurkers who were paid participants?

Read More