Jane McGonigal

Summary

This paper describes the concept of immersive gaming.  In order to convey this concept, the author gives an example of an online group of gamers known as the Cloudmakers.  This online group of gamers were a group of people who enjoy games that involve solving puzzles.  As described in the paper, this group proudly adapted the theory of being a collective detective who employed all of the resources at their disposal to solve any mystery/puzzle that was presented to them (no matter how obscure).  It was around this time that a massive immersive game was created that catered to groups of people with similar interests: The Beast.  This game created an effective means of virtual immersion.  The entire point of this game was to make it as close to reality as possible.  The creators of this game went as far as denying the existence of the game itself in order to promote its underlying theme of a conspiracy.  This game’s popularity was stemmed from the fact that it went beyond strictly online gaming and offline lives of its players in order to promote the augmented reality of this game.    It facilitated the need for collaboration among all of its players because it created such a complex network of puzzles that one person alone could not possibly solve all the problems.

Next, this paper gave a brief section on the difference between immersive and pervasive gaming.  Although they have many similar characteristics, these two differ in one fundamental manner: immersive games attempt to disguise the existence of the game to create a more realistic sense of a conspiracy, whereas a pervasive game is promoted and openly marketed to gain attraction.  In addition, immersive games encourage collaboration whereas the (Nokia Game) provided incentives to solvers of the game (which implicitly limited collaboration).  The Beast was a very complex network of puzzles, whereas the Nokia Game was simple enough that a single player could solve the game.

This paper then states some of the side effects of creating such immersive games.  It briefly tells of games as becoming too addictive and could potentially harm peoples’ lives.  However, it also emphasizes the players’ burning desire to keep the game play going and them consistently trying to make a conspiracy when one simply does not exist.  It makes the case that if these players have such a burning desire to solve complex puzzles, why not utilize their expertise and intelligence on real world problems?  Instead of fabricating conspiracies, why not apply them to the problems that governments currently face in order to come up with solutions?

Reflections

This paper was very interesting because I didn’t know that communities such as this existed.  I have heard of clans and groups forming in MMORPG’s such as World of Warcraft, however, never a game whose sole purpose was to be disguised so much as to make players question whether it was “not just a game”.  I appreciate the fact that the author pointed out some of the downfalls of this type of gaming.  These types of games can become highly addictive and cause massive amounts of personal damage to the gamers’ lives.  In addition, it can help to create a sense of paranoia to an already flustered society that we currently live in.   The fact that players are so willing to jump into the flames to solve any problem that is being thrown at them means that they may be manipulated at any point to solve real world problems without them ever knowing it.  However, this seems to be a double-edged sword.  If communities such as the Cloudmakers were put to solve a real-world task, they might stumble upon something that was not meant for the public, causing mass hysteria and/or as Rheingold stated: create a mob mentality.  I know that this example is a bit of a stretch, but one could almost consider Anonymous roughly similar to Cloudmaker.  They are a group of hackers/activists who are actively working on solving a problem and/or uncovering some truth that is meant to stay hidden.  I believe that if we were to employ these type of games, it could quickly turn into a form of attack.  For example, if there was a task published to hack into company X’s website (part of the game), and the players succeeded, this could potentially cause much harm to the company.  But who would be the person to get blamed? Would it be the person who got tricked into hacking the website in the first place or the pseudo game designer that left a vague clue that may or may not be interpreted.  This paper stated that the online community is very intelligent and that it greatly surpassed the game-maker’s expectations, if this intelligence was put to malicious use, it could have some potentially disastrous results. A great example of this could be the users of Reddit who falsely accused someone of being behind the Boston bombing.

Questions

1. How do you draw a line to distinguish game from reality
2. Should such an addicting type of game be banned
3. Is it wise to employ online intelligence to solve sensitive problems
4. Wouldn’t this create a constant sense of paranoia and eventually lose faith in the government?

 

Read More

Authors: Firas Khatib, Seth Cooper, Micheal D. Tyke, Kefan Xu, Ilya Makedon, Zoran Popvic, David Baker

Discussion Leader: Divit Singh

Crowdsourcing Example: http://weathersignal.com

Summary

Foldit is an online puzzle video game.  It presents a platform on which multiple players can collaborate and compete on various tasks such as protein folding.  It utilizes citizen science: leveraging natural human abilities for scientific purposes.  Foldit provides the players with a palette of interactive tools and manipulations to aid them in structuring the protein presented to them.  In addition, Foldit also provides players with the ability to create their own “recipes” for augmenting proteins.  These recipes are a set of instructions and game play macros that enable the players using the recipe to automatically manipulate the proteins presented to them.  User-friendly algorithms from the Rosetta structure prediction methodology were presented as well to aid players in interacting with structures.  From observing how players utilized these algorithms, it became apparent that the players used these algorithms to aughment rather than to substitute for human strategizing.   There was no one algorithm that was employed.  At different stages of interaction, players would use multiple recipes to build their structures which in turn, lead to more recipes being created.

During the time of the study, researchers created the “Fast Relax” algorithm which achieved better results in less time.  However, an algorithm was also developed by the Foldit players during this time: “Blue Fuse”.  These algorithms were very similar to each other and developed completely independently.  On testing these algorithms in side by side, it was revealed that Blue Fuse is more effective than Fast Relax (in Foldit) on time scales best compatible with game play.  The discovery of this algorithm was created solely by Foldit players.

Reflection

This paper is about a popular crowdsourcing framework used in the bioinformatics field.  It presents a unique way to utilize the brainpower of the general masses to create efficient  new and efficient algorithms by introducing a gaming aspect to protein folding.  I really liked how they allowed the players to build their algorithsm/simulations by employing the concept of “recipes”.  I believe that this was a crucial feature that allowed other players to build off someone else’s work rather than starting from scratch and coming up with either their own small contribution or replicating someone else’s work.  They present a clear UI with a helpful suite of tools to help in manipulating the structure as well.  In addition, I found that there were videos on YouTube as well as abundant information on their website to really emphasize the purpose of this software.

Figures 3 and 4 really emphasized the power of citizen science as it shows the social evolution of Foldit recipes.  New recipes are essentially built on top of each other in hopes to gain marginal efficiency with each iteration.  Instead of using machine learning in an attempt to approximate these recipes and simulations, real humans creations were used to develop algorithms.  The fact that these recipes resembled that of an algorithm produced by researchers specifically focused on producing an efficient algorithm shows the power of human computation.  As it stands, machine learning can only take us so far, especially in visual tasks such as these.

Questions

1. What are your opinions on gamifying problem solving/reasoning tasks such as this to attract a crowd?  Do you think it takes away from the task at hand by attracting a crowd that may be too young/old for its purpose? If so, how would you leverage gamification/any other task to try to attract the specified target audience?

2. Assuming there was no “energy” function in which to rate recipes for.  Based on visual aesthetics, how would you create a metric to measure how “clean” or “well-produced” a certain recipe is?

3. Would you rather have recipes be built on top of each other, or have individuals try to create their own from scratch? If you want them to be built on top of each other, does it not “tunnel-vision” subsequent creators?

Read More

Lydia Chilton, Juho Kim, Paul André, Felicia Cordeiro, James A. Landay, Daniel S. Weld, Steven P. Dow, Robert C. Miller, Haoqi Zhang

Discussion Leader: Divit Singh

Summary

In a conference, similar papers are usually organized into sessions.  This is done so that conference attendees can see related talks in the same time-block.  The process of organizing papers into these sessions is nontrivial.  This paper offers a different approach in order to aid the process of grouping papers into sessions.  This paper presents Frenzy: a web application designed to leverage the distributed knowledge of the program committee to rapidly group papers into sessions.  This application breaks session-making into 2 sub-problems: meta-data elicitation and global constraint satisfaction.  In the meta-data elicitation stage, users search for papers via queries on their abstracts/authors etc. and group them into categories that they believe makes sense.  They also have the ability to “+1” categories that have been suggested by other users to show support for that category.   In the global constraint satisfaction stage, users must assign a paper to a session and also make sure that every session contains at least two papers in it.  The author(s) tested this application at CSCW 2014 PC meeting and the schedule produced for the CSCW 2014 was generated with the aid of Frenzy.

Reflection

The idea of leveraging parallelism to create sessions for a conference is a brilliant one.  This paper mentioned that this process used to take an entire day and that the even then, the schedulers were usually pidgeon-holed into deciding which session a paper belonged to (in order to fulfill constraints).  By creating a web application that allows all users access to the data, I believe that they created an extremely useful and efficient system.  The only downside I see to this application is that I fear that they may give users too much power.  For example, users may delete categories.  I’m not sure that giving users this type of power would be wise.  For the purposes of a conference, where all users are educated and have a clear sense of the goal, it may be okay.  However, if they were to open up this system to a wider audience, this system may backfire.

I really liked how they divided up their process into sub-problems.   From my understanding, the first stage is to get a general sense as to where these papers belong and to get some user feedback to show where the majority of users believe the appropriate category for a paper should be.  This stage is open to the entire audience so that everyone may contribute and have a say.  The second stage is thought to be more of a “clean-up” stage.  A special subset from the committee members then make the final choices as to deciding papers for session.  Now, they are provided with the thoughts of the overall group, which greatly help in deciding where papers go.  In my head, I viewed this approach as a map-reduce job.  The metaphor may be a stretch, but I viewed the first stage, they are just trying to “map” a paper to the best possible category.  This task happens in parallel and it generates an increasing set of results.  The second stage, “reduces” these sets and delegates them into their appropriate sessions.  For those reasons, reading through this paper, it was pretty interesting how they were able to pull this off.  Apart from the information-intense UI that they provided for their web application, they did an excellent job in simplifying the tasks enough to produce valid results.

Questions

• The interface that Frenzy has contains a lot of jam-packed information.  Do you think as a user of this system, you would understand everything that was going on?
• The approach used by Frenzy breaks the problem of conference session making into 2 problems: meta-data elicitation and session constraint satisfaction.  Do you think that these two problems are to broad and can be broken down into further sub-problems?
• This system gives the power to “delete” categories.  How do you make sure that a category that is valid is not deleted by a user?  Can this system be used on a group that is larger than a conference committee? Ex: MTurk?

Read More

Eyal Peera , Sonam Samatb , Laura Brandimarteb & Alessandro Acquistib

Discussion Leader: Divit Singh

Summary

This paper focused on finding alternatives to MTurk and evaluating its results.  MTurk is considered to be the front-runner among other, similar crowdsorucing platforms since it tends to produce high quality data.  However, since the worker growth of MTurk is starting to stagnate, the workers that use MTurk have become extremely efficient on completing the tasks that are often published on MTurk.  The reason is because these tasks tend to be very similar (surveys, transcribing etc).  The familiarity with tasks have shown to reduce effect sizes of research findings (completing same survey multiple times with same answers skews data to be collected).  Due to this reason, this paper explores other crowdsourcing platforms and evaluates the performance, results, similarities and differences between each other in an effort to find a viable alternative for researchers to publish their tasks.

In order to evaluation the performance of these various crowdsourcing platforms, they tried to create a survey among all 6 of the platforms being tested.  Among these 6, only 3 of them successfully published the survey.  Some platforms simply rejected the survey without a reason, other platforms either required considerable amount of money or  had errors in their platforms which prevented the study from exploring those alternatives.  From the the platforms that were able to be publish the survey, it appeared that the only viable alternative to MTurk among the ones that were tested turned out to be CrowdFlower.  The surveys used involved questions that contained attention-check questions, decision-making questions, as well as a question which measured the honesty of the worker.   This paper provides an excellent overview of the various properties between each of the platforms and includes many tables which outline when one platform may be more effective over another.

Reflection

This paper does present a considerable amount of information among the various platforms that were described.  However, reading through this paper, it really revealed the lack of any actual competition to MTurk that is out there.  Sure, it does discuss that CrowdFlower is a good alternative in order to reach a different population of workers, it is still considered less than equal to MTurk for a lot of instances.  The main basis of using these other platforms is because MTurk workers have become extremely efficient at completing tasks which may cause the skewing of results.   I believe it is only a matter of time before workers on these other platforms lose their “naivety” as the platform becomes more mature.

The results of this paper may be invaluable to a researcher who wants to really target their audience.  For example, this paper revealed that CBDR is managed by CMU and that it is composed of students and non-students.  Although not guaranteed, it might be the most appealing for a researcher who wants to target college students since it may contain a considerable university student population.  Another excellent bit of information that they provided is the failure rate of attention-seeking questions that were posted on their survey.  This outlines two things: how inattentive workers are during their surveys, and also how experienced the workers of MTurk really are (they most likely have seen questions like these in the past which prevents them from making the same mistake again).  However, keep in mind that these results are a snapshot at a given time.  There is nothing that is prevented the workers of CrowdFlower (which are apparently disjoint from workers of MTurk) which contain a massive worker base from learning from these surveys and become smarter workers.

Questions

1. Is there any other test that you believe that the study missed?
2. Based on the tables and information provided, how would you rank the different crowdsourcing platforms?  Why?
3. This paper outlined that outlined different approaches for these platforms (e.g. review committee that determines if a survey is valid).  What method do you agree with or how would you design your own platform in order to optimize quality and reduce noise?

 

Read More