{"id":392,"date":"2017-12-18T17:37:43","date_gmt":"2017-12-18T17:37:43","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724f17\/?p=392"},"modified":"2017-12-18T17:37:43","modified_gmt":"2017-12-18T17:37:43","slug":"assessing-graph-evaluation-in-a-citizen-science-context","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/2017\/12\/18\/assessing-graph-evaluation-in-a-citizen-science-context\/","title":{"rendered":"Assessing Graph Evaluation in a Citizen Science Context"},"content":{"rendered":"

Need<\/strong><\/p>\n

A common task in computational biology is the creation of protein graphs to convey an idea.\u00a0 These ideas can range from showing molecular complexities of diseases [1] or general layout of cellular components [2].\u00a0 However, this is sometimes a complex task that can take a significant amount of time for an expert to complete.\u00a0 In previous work, the Crowd Intelligence Lab at VT has shown that crowdworkers on Amazon Mechanical Turk can create and evaluate this kind of graph.\u00a0 However, the field of citizen science (like the projects on Zooniverse [3]) has the potential to also be a good source of evaluation, which can allow a refinement feedback loop to create better graphs.<\/p>\n

 <\/p>\n

Approach<\/strong><\/p>\n

As previously mentioned, Zooniverse is a website that collects, hosts, and promotes citizen science projects.\u00a0 In addition, they have a good project wizard that helps budding project managers to easily get projects up and running.\u00a0 Therefore, I have taken a collection of 78 layouts in .jpg format and uploaded them to Zooniverse.\u00a0 Then, citizen scientists will execute a workflow that evaluates the graph along several metrics and prompts for qualitative feedback.\u00a0 A picture of the interface can be seen below.<\/p>\n

\"\"<\/p>\n

Benefit<\/strong><\/p>\n

This project aims to benefit biologists and related experts through giving them a tool to enhance and evaluate generated graphs.\u00a0 This tool should quickly give feedback on issues of aesthetics and readability, which will strengthen any argument they are trying to make with the graph.\u00a0 Furthermore, citizen scientists are performing analysis on these projects for intrinsic benefits rather than extrinsic benefits (like getting paid).\u00a0 Previous work has shown that difficult tasks are performed more precisely by citizen scientists than crowdworkers [4], which would should benefit this task.<\/p>\n

Competition<\/strong><\/p>\n

Competition comes from many places for crowdworkers in this field.\u00a0 Currently, there are 69 projects hosted on Zooniverse alone; each citizen science project vies for workers to analyze their data.\u00a0 Furthermore, this competes with the traditional methods of the expert just laying out the graph themselves and refining as needed.\u00a0 Some data is proprietary or has other needs to be kept private, so not all biologists will think this is an effective tool.\u00a0 Lastly, the CrowdLayout tool that was developed previously also competes; getting a crowdworkers to layout graphs in minutes is fairly effective.<\/p>\n

 <\/p>\n

Results<\/strong><\/p>\n

Due to Zooniverse\u2019s requirements on projects they promote, this experiment was unable to get to their main page.\u00a0 However, 161 responses were gathered after promotion via emails and flyers.\u00a0 Of these responses, 24 were discarded due to being incomplete answers.\u00a0 Quantitatively, the results are statistically similar after performing Mann-Whitney U-tests on each metric and the overall rating.\u00a0 Seen below are the boxplots of the data, showing that the averages and confidence.\u00a0 Qualitatively, I found that 62 of 119 responses (18 did not provide qualitative feedback) were constructive.\u00a0 This was described as containing some sort of placement or edge suggestions.\u00a0 Furthermore, there was evidence of problems with the interface (7 responses), malicious users (11 responses), users who didn\u2019t understand the task (8 users), and issues specifically with edge crossings (12 responses).<\/p>\n

\"\"<\/p>\n

Overall<\/strong><\/p>\n

\"\"<\/p>\n

Discussion<\/strong><\/p>\n

Zooniverse\u2019s interface and wizard can create a tool that allows for graph evaluation at a similar effectiveness as paid crowdworkers.\u00a0 Furthermore, the workers created action items in the form of constructive feedback in 52% of ratings.\u00a0 It was also odd to see malicious responses (as defined by Gadiraju et al. [5]), since these are volunteer-based studies.\u00a0 In addition, the hurdles that Zooniverse has for promotion on their website makes it difficult to serve as a permanent solution unless the throughput and graph generation increases.<\/p>\n

 <\/p>\n

Citations<\/strong><\/p>\n

\n
\n
\n
\n
\n
\n
    \n
  1. Baraba \u0301si, A.-L., Gulbahce, N., and Loscalzo, J. Network medicine: a network-based approach to human disease. Nature Reviews Genetics 12, 1 (2011), 56\u201368.<\/li>\n
  2. Barsky, A., Gardy, J. L., Hancock, R. E., and Munzner, T. Cerebral: a cytoscape plugin for layout of and interaction with biological networks using subcellular localization annotation. Bioinformatics 23, 8 (2007), 1040\u20131042.\n
    \n
    \n
    <\/div>\n<\/div>\n<\/div>\n<\/li>\n
  3. https:\/\/www.zooniverse.org\/<\/a><\/li>\n
  4. Mao, A., Kamar, E., Chen, Y., Horvitz, E., Schwamb, M. E., Lintott, C. J., and Smith, A. M. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing. In First AAAI conference on human computation and crowdsourcing (2013).<\/li>\n
  5. Gadiraju, U., Kawase, R., Dietze, S., and Demartini, G. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM (2015), 1631\u20131640.\n
    \n
    \n
    <\/div>\n<\/div>\n<\/div>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    Need A common task in computational biology is the creation of protein graphs to convey an idea.\u00a0 These ideas can range from showing molecular complexities of diseases [1] or general layout of cellular components [2].\u00a0 However, this is sometimes a complex task that can take a significant amount of time for an expert to complete.\u00a0 […]<\/p>\n","protected":false},"author":105,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-392","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/posts\/392","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/users\/105"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/comments?post=392"}],"version-history":[{"count":5,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/posts\/392\/revisions"}],"predecessor-version":[{"id":400,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/posts\/392\/revisions\/400"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/media?parent=392"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/categories?post=392"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724f17\/wp-json\/wp\/v2\/tags?post=392"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}