Analysis on Amazon Mechanic Turk and other paid crowdsourcing platform

Ever since the rise of AMT, research and applications have been focusing on this prominent crowdsourcing platform. With the development of similar platforms, some concerns on the use AMT have been solved in various ways. This paper reviews AMT limitations and compares the solution among 7 other popular platforms: ClickWorker, CloudFactory, CrowdComputing Systems, CrowdFlower, CrowdSource, LeadGenius and oDesk.

AMT Limitations are presented in four categories: They are short in quality control, management tools, support for fraud prevention and automated tools. These limitations are further mapped to assessment criteria to focus on detailed solution from all the platforms.

These criteria include the identity and skill of contributors, extra workload management, complex task support and quality control by requesters, and generalized qualification, task collaboration, task recommendation by platforms. By comparing with AMT, future focus research is addressed. Also, the method used in this paper can be improved in a way we set an alternative platform as a baseline.

I tried to work on a platform…

I’ve been thinking since I tried to work on Amazon Mechanic Turk and ClowdFlower (I believe they changed the name to Figure Eight). How does a requestor post a task? The expected format of the input and output from the platform may not match the interface provided by the requestor. I can see most requestors have to transform/code for themselves, but the platforms also start to help here.

Both platforms require identification through credit card and AMT requires SSN. I’m able to use Figure Eight now but AMT refused my signup. I have a relatively new SSN and credit record, which is probably the reason I’m refused. Although CrowdFlower comes into people’s sight and was mentioned more than before, the difference in scale and functionality can be easily spotted in their website layout and structure.

Figure Eight provides a basic task for me to start – A people’s name, current company and position name are given, my job is to go to his LinkedIn profile and make sure of two things: Is he/she working in the same company, and if so, did he/she change to another position? How many positions of this people are active in?  

This should be a simple task, even for a people who’s not familiar with LinkedIn. The reward is relatively low, though. For 10 correct answers I got 1 cent (I think I’m still in an evaluation period). More pay is on the road if I work in a recommended manner, i.e. to try out several simple tasks in different categories, and then put my hands on more complex task, and so on.

Still, I found myself quitting after I made 10 cents. I’m not sure if it’s because I was too casual in sample quiz that I got 8 correct out of 10 and that they decided to give me a longer trail. Compared to several fun-inspired tasks I tried, experience on Figure Eight is not so welcome as I see it.

Back to the analysis. There’s an example that represent many dilemmas in this tripartite workspace: AMT doesn’t care about workers’ profile while oDesk offers public worker profiles showing their identities, skills, rating, etc. It’s hard to maintain a platform that workers can switch between these two options when identification is preferred for some tasks and not for other tasks. And this may refrain requestors from posting their needs.

Questions

Do platforms cooperate? How to combine existing good solution to improve or come up with a better platform?

How does AMT dominate in crowdsourcing for so long and no other platform catches up with it? What are the most significant improvements on AMT in the recent years?

Read More

Beyond evaluating the extent of AMT’s influence for research, Vakharia’s paper Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms draws the impact of existing crowd work platforms based on prior work, comparing the operation and workflows among the platforms. In addition, it opens discussions about to what extent that the characteristics of other platforms may have positive and negative impact on the research. Built on analysis of prior white papers, this paper introduces inadequate quality control, inadequate management tools, missing support for fraud prevention, and lack of automated tools. Further, this paper defines twelve ways to comprehensively evaluate the performance of the platforms. At last, the researchers present a comparative results based on their proposed metrics.

While reading this paper, I am thinking that the private crowds problem is also an ethics problem because protection of sensitive or confidential data is significant to different parties. Besides, the ethics problems of the crowd work platforms can be seen from three perspectives, saying, data, human, and interfaces. From the data perspective, we need to ask the requester to provide the rational data by mitigating the bias, such as gender and geo-location. From the human  perspective, we need the platforms to assign the workers tasks randomly. From the interface perspective, the requesters need to provide the interfaces that is symetrics for all categories of data, and the data should follow the IID distribution.

Though I have not used a platform like AMT before, I performed data labeling tasks before. The automated tools are really efficient and important to the kind of worker like this. I was part of a team labeling cars for an object tracking project before. I used a tool to automatically generate the object detection results, but the results are highly biased. However, even if the data is biased, it still assisted us a lot in labeling the data. 

The paper provides a number of criteria to qualify the characteristics of various platforms. For the criterion “Demographics & Worker Identities”, it also needs an analysis on the other criterion, whether it is ethical to release the demographics of workers and requesters. What would be the potential hazard to make the personal information available? The two criterions “Incentive Mechanisms” and “Qualifications & Reputation” seem to have some conflicts with each other. Since if the workers work faster on the tasks, that would potentially affects the quality of the work. As for the quantify metrics, the paper does not provide quantitative metrics to quantify the performance of different crowd work platforms. Hence, it is still difficult for the users and requesters to judge which crowdsourcing platforms are good for themselves. The following questions are worthy of further discussion.

• What are the main reasons of crowd workers or requesters to pick one particular platform over others as their major platform? 
• Knowing the characteristics of these platforms, is it possible to design a platform which boasts of all the merits?
• A lot of criteria are provided in the paper to compare the crowd work platforms, is it possible to develop quantified standard metrics to evaluate the characteristics and performance?
• Is it necessary or is it ethical for the requesters to know the basic information of the workers, and vice versa?

Read More

Summary

In this paper, the authors aim to highlight and explore the features of online crowd works other than Amazon Mechanical Turk (AMT) that is not been investigated by many researchers. They recognize that AMT as a system that made a revolution on data processing and collection, but also lack very crucial features like quality control, automation, and integration.

The paper poses many questions about human computation and presents some solutions. The questions related to the current problems with AMT, features of other platforms, and a way to measure each system.

The authors discuss the limitation the AMT system like lacking a good quality control, lacking a good management tools, and missing a process to prevent fraud, and lacking of a way to automate repeated tasks.

The paper defines criteria to evaluate or asses a crowd work platform. The assessment uses different categories like incentive program, quality measures used in the system, the worker demographic and identity information, worker skills or qualifications, and other categories that they have used to compare/contrast different systems including AMT.

The authors also reviewed like seven AMT-alternatives like ClickWorker, CloudFactory, CrowdComputing Systems, CrowdFlower, CrowdSource, MobileWorks, and oDesk. They show the benefits of each system over AMT using the criteria mentioned above, and show that there is a significant improvement on these systems that make the entire process much better and enable the worker and the requester to interact in a better way.

The crows-platforms analysis done by the authors was the only work at the time the paper was written to compare different systems on a defined criterion, and they hope that this work offer a good foundation for other researchers to get better results.

All platforms are still missing features like analytics data about each worker to provide visibilities of the work that is done by each worker. There is also a lack of security measure to make sure that the system is robust and can respond to any adversarial attack.

Reflection

I found that the features that are missing from Amazon Mechanical Turk interesting giving the volume of the system usage and also given the work that Amazon do today on the cloud computing, marketplace and other area where the quality of the work is well known.

I also found that the technical details mentioned in the paper interesting. It seems to me that the authors go lots of feedback from everybody get involved in the system.

I agree with authors on the criteria mentioned in the paper to asses crowdsourcing systems like pay motivation, quality control, and automation and system integration.

The authors didn’t specify which system is the best on their opinion and which system meet all of their criteria.

Questions

• Are there any other criteria that we can use to assess the crowdsourcing systems?
• The author didn’t mention which system is the best. Is there a system that can outperform others?
• Is there a reason why Amazon didn’t address these findings?

Read More

Summary:

In this paper, the authors analyze and compare different crowd work platforms. They comment that research into such platforms has been limited to Mechanical Turk and their study wishes to encompass more of them. 

They compare seven AMT alternatives namely ClickWorker, CloudFactory, CrowdComputing Systems, CrowdFlower, CrowdSource, MobileWorks, and oDesk. They evaluate the platforms on 12 different metrics to answer the high-level concerns of quality control, poor worker management tools, missing fraud prevention measures and lack of automated tools. The paper also distinctly provides a need from requesters to employ their own specialized workers through such platforms and apply their own management systems and workflows. 

The analysis shows diversity of these platforms and identifies some commonalities such as “peer review, qualification tests, leaderboards, etc.”  and also some contrastive features such as “automated methods, task availability on mobiles, ethical worker treatment, etc.”

Reflection:

The paper provides great evaluation metrics to judge aspects of a crowd work platform. The suggested workflow interfaces and tools can greatly streamline the process for requesters and workers. However, I don’t think these crowd work platforms are businesses. Hence, incentive is required to invest in such additional processes. In the case of MT, the competitors do not have enough market share to promote viability of additional streamline processes. I think as the processes become more complex, requesters will be limited by the current framework and a market opportunity will force the platforms to evolve by integrating the processes mentioned in the paper. This will be a natural progress based on traditional development cycles.

I am sure a large company like Amazon definitely has the resources and technical skills to lead such a maneuver for MT and other platforms will follow suit. But the most important aspect for change would be a market stimulus driven by necessity and not just desire. Currently, the responsibility falls on the requester because the requirement for the processes is rare.

Also, the paper only analyzes from a requester perspective. Currently, the worker is just a de-humanized number but adding such workflows may lead to discrimination between geographical regions or distrust in a worker’s declared skill sets. This will bring the real-world challenges in the “virtual workplace” and more often lead to challenging work conditions for remote workers. This condition might also lead to worrisome exclusivity which the current platforms avoid really well. However, I believe user checks and fraud networks in the system are areas that the platforms should really focus to improve user experience for requesters.

I think a different version of the service should be provided to corporations who need workflow management and expert help. For quality control, I believe the research community should investigate globally applicable efficient processes for these areas.

Questions:

1. How big is the market share of Mechanical Turk compared to other competitors?
2. Does Mechanical Turk need to take a lead in crowd work reforms?
3. Is the difference between platforms due to the kind of crowd work they support? If so, which type of work has better worker conditions?
4. How difficult is for MT integrate the quality controls and other challenges mentioned in the paper?

Read More

Summary:
“Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms” by Donna Vakharia and Matthew Lease talks about the limitations of Amazon Mechanical Turk (AMT) and presents a qualitative analysis of newer vendors who offer different models for achieving quality crowd work. AMT was one of the first platforms to offer paid crowd work. However, nine years after its launch, it still is in its beta stage because of its limitations that fail to take into account the skill, experience and ratings of the worker and the employer and its minimal infrastructure that does not support collecting analytics. Several platforms like ClickWorker, CloudFactory, CrowdComputing Systems (now WorkFusion), CrowdFlower, CrowdSource, MobileWorks (now LeadGenius), and oDesk have tried to mitigate these limitations by coming up with a more advanced workflow that ensures quality work from crowd workers. The authors identify four major limitations of AMT: (i)Inadequate Quality Control, (ii)Inadequate Management Tools, (iii)Missing support for fraud prevention, and (iv)Lack of automated tools. The authors also list down several metrics to qualitatively assess other platforms: (i)Key distinguishing features of the platform, (ii)Source of the workforce, (iii)Worker demographics, (iv)Worker Qualifications and Reputations, (v)Recommendations, (vi)Worker Collaboration, (vii)Rewards and Incentives, (viii)Quality Control, (ix)API offerings, (x)Task Support and (xi)Ethics and Sustainability. These metrics prove useful for a thorough comparison of different platforms.

Reflections:
One of the major limitations of AMT is that there are no pre-defined tests to check the quality of the worker. In contrast, other platforms ensure that they test their workers in one or the other way before assigning them tasks. However, these tests might not always reflect the ability of the workers. The tests need to be designed keeping the task in mind and this makes standardization a big challenge. Several platforms also believe in offering their own workforce. This can have both positive and negative impacts. The positives being that the platforms can perform a thorough vetting while the negatives are that this might limit the diversity of the workforce. Another drawback of AMT is that workers seem interchangeable as there is no way to distinguish one from the other. Other platforms try to use badges to display worker skills and use a leaderboard to rank their workers. This can lead to unequal distribution of work which might be merit based but there is a need to perform a deeper analysis of the ranking algorithms in order to ensure that there is no unwanted bias in the system. Some of the platforms employ automated tools to perform tasks which are repetitive and monotonous. This comes with its own set of challenges. As machines become more “intelligent”, humans need to develop better and more specialized skills in order to remain useful. More research needs to be done in order to better understand the working and limitations of such “hybrid” systems.

Questions:
1. With crowd platforms employing automated tools, it is interesting to discuss whether these platforms can still be categorised as a crowd platform.
2. This was more of a qualitative analysis of crowd platforms. Is there a way to quantitatively rank these platforms? Can we use the same metrics?
3. Are there certain minimum standards that every crowd platform should adhere to?

Read More

The authors found that most of current research on crowd work focused on AMT, and there is little exploration of other platforms. To enrich the diversity of crowd work researches and accelerate progress of development of crowd work, the authors contrast the key problems which they have found in AMT versus features of other platforms. With the comparison, the authors found different crowd platforms provide solution to different problems in AMT. However, all the platforms cannot provide sufficient support to well-designed worker analytics. Additionally, they made a conclusion that there is still a number of limitations in AMT and research on crowd work will benefit from diversifying. With the contributions of the authors, future research on alternative platforms can benefit from not considering AMT as the baseline.

Reflection:

AMT is the first platform for crowd work and many people are benefit by using this platform. However, if all companies and workers only use AMT, the lack of diversity will result in a low development progress of crowd work platform.

From AMT’s point of view, if there are no other crowd work platforms, it is hard for AMT to effectively find solutions to its limitations and problems. In this paper, for example, lack of automated tool is one of AMT’s limitations. However, the platform WorkFusion allows usage of machine automated workers, which is an improvement to AMT. If there is only one platform, it is hard for the platform to find its limitation by itself. But with the competitor platforms, to provide better user experience and defeat other adversary companies, they have to do a lot of research to keep their platforms up to date. As a result, the research of crowd work will be pushed to develop rapidly.

For other platforms, they should not just copy the pattern of AMT. Though AMT is the most popular crowd work platform, it still has some limitation. Other platforms can copy the advantages of AMT. However, they should avoid AMT’s limitations and find solutions to these drawbacks. A good baseline can help the initiation of other platforms. But if other platforms are just totally follow the baseline, all the platforms will suffer from same drawbacks and no one can propose any solution to them. To avoid this kind of situation, the companies should develop own platforms, avoid the drawbacks and learn advantages from other platforms.

The researchers should not just focus on AMT. Of course, the researches on AMT will receive more attention as most of the users use AMT. However, it is harmful to the development of crowd work platform. Even some platforms developed solutions for some problems, if the researchers just ignore these platforms, these solutions will not be able to spread out and the researchers and other companies will spend unnecessary effort to do research on similar solutions.

Question:

What is the main reason for most of the companies select AMT as their crowd work service provider?

What is the most significant advantage for each of the platforms? Is there any chance to develop a platform which has all other platforms’ advantages?

Why most of the researchers focused on AMT only? Is it possible to advice more researchers to do cross-platform analysis?

Read More

Paper Authors: Donna Vakharia and Matthew Lease.

Summary

This paper gives a general overview of different crowdsourcing platforms and their key feature offerings, while centering around the limitations of Amazon Mechanical Turk (AMT), the most popular among the platforms. The factors which make requesters resort to AMT are briefly discussed, but the paper points out that these factors are not exclusive to AMT. Other platforms also offer most of these advantages, while offsetting some of AMT’s limitations such as quality control, automated task routing, worker analytics, etc.. The authors qualitatively assess these platforms, by comparing and contrasting on the basis of key criteria categories. The paper, by providing exposure to lesser-known crowdsourcing platforms, hopes to mitigate one plausible consequence of researchers’ over-reliance on AMT i.e. the platform’s limitations can sub-consciously shape research questions and directions. 

Reflection

1. Having designed and posted a lot of tasks (or HITs) on AMT, I concur with the paper’s assessment of AMT’s limitations, especially no built-in gold standard tests, no support for complex tasks, task routing and real-time work. The platform’s limitations, essentially, is offloaded by the researcher’s time, efforts and creativity, which is now consumed to work around these limitations instead of other pressing stuff.
2. This paper provides a nice exposure to platforms that offer specialized and complex task support (e.g. CrowdSource supporting writing and text creation tasks). As platforms expand on supporting for different complex tasks, this would a) reduce the workload on requesters for designing tasks, and b) reduce the quality control tensions arising from poor task design.
3. Real-time crowd work, despite being an essential research commodity, still remains a challenge for crowdsourcing platforms. Even though this inability has resulted in toolkits like LegionTools [1] which facilitate real-time recruiting and routing of crowd workers on AMT, these toolkits are not the final solution. Even though many real-time crowd-powered systems have been built using this toolkit, they still remain prone to being bottle-necked by the toolkit’s limitations. These limitations may arise from lack of resources for maintaining and updating the software, which may have originated as a student-developed research project. Crowd platforms adopting such research toolkits into their workflow may solve some of these problems. 
4. Sometimes, projects or new interfaces may require testing learning curve of its users. It does not seem straightforward to achieve that on AMT since it lacks support for maintaining a trusted worker pool. However, it seems to be possible on other platforms like ClickWorker and oDesk, which allow worker profiles and identities.  
5. A new platform, called Prolific, was launched publicly in 2019, and alleviates some of the shortcomings of AMT such as fair pay assurance (with a minimum $6.50/hour), worker task recommendation based on experience, initial filters, and quality control assurance. The platform also provides functionalities for longitudinal/multi-part studies, which may seem difficult to achieve using the functionalities offered by AMT. The ability for longitudinal studies was not addressed for other platforms, either. 
6. The paper was published in 2015 and highlighted the lack of automated tools. Since then, numerous services have come up and now offer human-in-the-loop functionalities, including Amazon and Figure Eight (formerly CrowdFlower). 

Questions

1. The authors raise an important point that the most popularly used platform’s limitations can shape research questions and directions. If you were to use AMT for your research, can you think of how its shortcomings would affect your RQs and research directions? What would be the most ideal platform feature for you?
2. The paper advocates algorithms for task recommendation and routing, as has been pointed out in other papers [2]. What are some other deficiencies that can be supported by algorithms? (reputations, quality control, maybe?)
3. If you had a magic tool to build a crowdsourcing platform to support your research, along with bringing a crowd workforce, what would your platform look like (the minimum viable product)? And who’s your ideal crowd? Why would these features help your research?

Read More

SUMMARY

In this work, the authors perform a study that extends to crowd work platforms beyond Amazon’s Mechanical Turk. They state that this is the first research that has attempted to perform a more thorough study of the various crowd platforms that exist. Given that prior work has mainly focused on Mechanical Turk, a large number of the issues faced by both requesters and workers has been due to the idiosyncrasies associated with this platform in particular. Thus, the authors aim to broaden the horizon for crowd work platforms in general and present a qualitative analysis of various platforms such as ClickWorker, CloudFactory, CrowdComputing Systems, CrowdFlower, CrowdSource, MobileWorks, and oDesk. The authors identify the key criteria to distinguish between the various crowd platforms as well as identify key assumptions in crowd sourcing that maybe caused due to a narrow vision of AMT.

The limitations of AMT, as described by the authors, include inadequate quality control (caused due to a lack of gold standards, lack of support for complex tasks), inadequate management tools (caused due to lack of details about a worker’s skills, expertise; lack of focus on worker ethics and conditions), missing support to detect fraudulent activities, and a lack of automated tools for routing of tasks. In order to compare and assess the seven platforms selected for this study, the authors focus on four broad categories – quality concerns, poor management tools, missing support to detect fraud, and lack of automated tools. These broad categories further map to various criteria such as identifying distinguishing features of each platform, identifying if the crowd platform maintains its own workforce or relies on other sources for its workers as well as if the platform allows for/offers a private workforce, the amount and type of demographic information provided by the platform, platform support for routing of tasks, support for effective and efficient communication among workers, the incentives provided by the platform, organizational structures and processes for quality assurance, existence of automated algorithms to help human workers, and the existence of an ethical environment for workers.

REFLECTION

I found it interesting to learn that prior research in this field was done mainly using AMT. I agree that the research that was performed with only AMT as the crowd platform would have led to conclusions that were biased due to this narrow vision of crowd platforms in general. I believe that the qualitative analysis performed by this paper is an important contribution to this field at large as it will help future researchers to select a platform that is best suited for their task at hand due to a better awareness of the distinguishing features of each of the platforms considered in this paper. I think that the analogy about the Basic programming language aptly describes the motivation for the study performed in this paper.

I also found the categories selected by the authors to be interesting and relevant for requesters when they are considering a platform to be chosen. However, I think that it may have also been interesting (as a complementary study) for the authors to have included information about reasons why a crowd worker may join a certain platform – this would give a more holistic perspective and an insight into these crowd working platforms beyond Mechanical Turk. For example, the book on Ghost Work, included information about platforms such as Amara, UHRS, and LeadGenius in addition to AMT. Such a study coupled with a list of limitations of AMT from a workers’ perspective as well as a similar set of criteria for platform assessment would have been interesting.

QUESTIONS

1. Considering that many of the other crowd platforms such as CrowdFlower (2007), ClickWorker (2005) have existed before the date of publication of this work, is there any specific reason that prior research with crowd work did not explore any these platforms? Was there some hinderance to the usage and study of these platforms?
2. The authors mention that one of the limitations of AMT was its poor reputation system – this made me wonder why AMT did not take any measures to remedy this poor reputation system?
3. Why is that AMT’s workforce is focused in U.S. and India? Do these different platforms have certain distinguishing factors that cause certain demographics of people to be more interested in one over the other?
4. The paper mentions that oDesk provides payroll and health-care benefits to its workers. Does this make requesting on oDesk more expensive due to this additional cost? Are requesters willing to pay a higher fee to ensure such benefits exists for crowd workers?

Read More

Summary:

Since the AMT has long been the research focus for the study of crowdsourcing. The author compared a series of platforms and their diverse capabilities to enrich future research directions. The aim of his work is to encourage more investigation and research studies from different sources instead of focusing only on AMT. The author reviewed the related work, pointed out the AMT limitation and problems that have been criticized, then defined the criteria to assess the platform. Mainly in the paper, the author performed a detailed analysis and comparison among seven alternative crowd work platforms. Through the analysis and comparison with AMT, the author identified the same approaches that each platform has been implemented, such as the peer assessment, qualification tests, leaderboards, etc. Besides, the author also found the difference such as the use of automated methods, task availability on mobiles, ethnic worker treatment, etc.

Reflections:

I think the most serious problem in the research study is the limitation of the scope of the investigation. Only a variety of usage of resources can make the research results credible and highly applicable. Thus, I think it is very meaningful for the author to make this comparison and investigation. This is what research should so, always explore, always compare and keep critical. Besides, with the increase in popularity and the increase in the number of users, AMT should pay more attention to improving its own platform, rather than keep staying at a level that can meet current needs. Especially now that the same type of platforms are gradually increasing and developing, they are attracting more users by improving a better management service or develop reasonable regulation and protection mechanisms. Although, until now, AMT is the biggest and most developed platform, AMT still needs to learn from other platforms’ advantages.

In spite of that, other platforms should also keep update and try to create novelty features to attract more network users. A good way to improve their own platform is to always consider what the user’s requirement is. Hot topics which include ethical issues and labor protection issues should be considered. Besides, how to make good use of these platforms to a great extent to improve their product quality is also worth considering.

A short path towards improvement is through discussion. This discussion should include the company, the client, the product development team and even the researcher. As for the companies, they should always ask feedback from their network users. This is a baseline for them to improve not only their platform and user experience but also their product. Also, companies should discuss with each other and even though with the researchers to think about solutions to current problems in their platforms. Even though we cannot predict how things will go. Will these platforms last long? How many of these works will be available with the development of technology? I still believe it is worth to work hard to make these platforms better.

Questions

• Why researchers are more willing to do research on AMT?
• Is there any solution to pursue researchers do research on other platforms?
• For the other platforms besides AMT, what is the main reason for their users to choose them instead of AMT?

Read More

In 2005, Amazon launched an online crowd work platform named Mechanical Turk which was one of the first of its kind and it gained momentum in this space. However, numerous other platforms have come up offering the same service since then. A vast majority of researchers in the field of crowd work have concentrated their efforts on Mechanical Turk and often ignore the alternative feature sets and workflow models provided by these other platforms. This paper deviates from this pattern and gives a qualitative comparison of 7 other crowd work platforms. The intent is to help enrich research diversity and accelerate progress by moving beyond MTurk. Since there has been a lot of work inspired by the short-comings of MTurk, the broader perspective is often lost and the alternate platforms are often ignored. This paper covers the following platforms: ClickWorker, CloudFactory, CrowdComputing Systems, CrowdFlower, CrowdSource, MobileWorks, and oDesk.

I feel that this paper encompasses different types of crowd work platforms and provides a holistic view as opposed to just focusing on one platform. The different dimensions used to compare the platforms give us an overview of the differentiating features each platform provides. I agree with the author in that research on crowd work would benefit from diversifying its lens of crowd work. This paper would be a good starting point from that perspective.

Having only been exposed to MTurk and its limitations thus far, I was pleasantly surprised to note that many platforms offer peer reviews, plagiarism checks, and feedback. This not only helps ensure a high quality of work but also provides a means for workers to validate their work and improve. Opportunities are provided to enhance the skill set of workers by providing a variety of resources to train them like certifications and training modules. Badges are used to display the workers’ skill set. This helps promote the worker’s profile as well as helps the worker grow professionally. Many platforms display work histories, test scores, and areas of interest that guide requesters in choosing workers who match their selection criteria. A few platforms maintain payroll and provide bonuses for high performing workers. This keeps the workers motivated to deliver high-quality results. 

I really liked the fact that a few platforms are using automation to complete mundane tasks thereby eliminating the need for human workers to do these tasks. These platforms identify tasks that can be handled by automated algorithms, use machine automated workers for these tasks and use human judgment for the rest. This increases productivity and enables faster completion times.

• How can the platforms learn from each other and incorporate the best practices so that they can provide a platform that motivates the workers to perform well as well as helps requesters find workers with the necessary skillset efficiently?
• What are some ways we can think of that permits access to pull identities for greater credibility and hide identities when not desired? Is there a way a middle ground can be achieved?
• Since learning is an extremely important factor that would benefit both the workers (professional growth) and requesters (workers are better equipped to handle work), how can we ensure that due importance is given to this aspect by all platforms?

Read More