Summary
The paper by Hara et al. attempts to address the problem of sidewalk accessibility by using crowd workers to label the data. The authors had different contributions in addition to just making crowd workers label images. They conduct two studies, a feasibility study and an online crowdsourcing study using AMT. The first study aims to find out how practical it is to label sidewalks using reliable crowd workers (experts). This study also gives an idea of the baseline performance and acts as a validated ground truth data. The second study aims to find out the feasibility of using Amazon Mechanical Turks for this task. They have evaluated the accuracy of image-level as well as pixel-level. The authors have conducted a thorough background study on the current sidewalk accessibility issues, the current audit methods, and that of crowdsourcing and image labeling. They were successful in showing that untrained crowd workers could identify and label sidewalk accessibility issues correctly in the google street view imagery.
Reflection
Combining crowdsourcing and google street view to identify street- level accessibility is essential and useful for people. The paper was an interesting read and the authors described the system well. In the video[1], the authors show the instructions for the workers. The video gave a fascinating insight into how the task was designed for the workers, explaining every labeling task in detail.
The paper mentions accessibility, but they have restricted their research for wheelchair users. This works for the first study as they are able to label the obstacles correctly, and this gives us the ground truth data for the next study as well as establishes the feasibility of using crowd workers to identify and label accessibility effectively. However, accessibility problems on sidewalks are also faced by other groups like people with reduced vision, etc. I am curious to see how the experiments would differ if the user-group and the need changes?
The experiments are based on google street view, which is not known to be the best at certain times. There are certain apps that help people get real-time updates on traffic while driving like the app Waze [2]. I was wondering if google maps or any other app insert dynamic updates for street walks, it would be beneficial. It would not only help people but also help the authority in determining which sidewalks are frequently used and the most common issues people face. The paper is a bit old. But, newer technology would surely help users. The paper [3] by the same author is a massive advancement in collecting sidewalk accessibility data. This paper is a good read based on the latest technology.
The paper mentions that active feedback to crowd workers would help improve labeling tasks. I think that dynamic, real-time feedback would be immensely helpful. However, I do understand that it is challenging to implement when using crowd workers, but an internal study could be conducted. For this, a pair or more people need to work simultaneously, where one label and the rest give feedback or some other combinations.
Questions
- Sidewalk accessibility has been discussed for people with accessibility problems. They have considered people in wheelchairs for their studies. I do understand that such people would be needed for study 1, where labeling is a factor. However, how does the idea extend to people with other accessibility issues like reduced vision?
- This paper was published in 2013. The authors do mention in the conclusion section that with improvement in GSV and computer vision will overall help. Has any further study been conducted? How much modification of the current system is needed to accommodate the advancement in GSV and computer vision in general?
- Can dynamic feedback to workers be implemented?
References
Bipasha, great point! I have the similar comment on the groups selected for this study. As far as I am concerned, accessibility has a large scope, but the researchers narrow that down to merely focus on one facet of the problem specifically for a specific group (wheelchair users). However, the potential accessibility issue other groups (e.g. the elder with walking stick, individuals with reduced vision). Those groups may have different concerns, such as pavement design and audio assistant crosswalk. To identify more complex accessibility problems, it may require street imagery of good quality captured by desirable focal point of camera. Also, the groups recruited in the study need to be trained with basic background knowledge of sidewalk design. It would be helpful to get urban planners specified in streetscape design involved in the study.
It depends on the task, but dynamic feedback seems feasible (definitely challenging!). Feedback can be provided at the system/interface-level (“Hey, congrats! you completed the first batch of images” or something like that) or task-level (“Incorrect label provided”). The former is more on the UX side of things, while the latter can be challenging. Maybe sneaking in gold-standard data could work. Or if you have a really good prediction algorithm, that could provide feedback to the workers. But then, you will have to ensure that the algorithms are reliable enough to provide feedback. Or. you could show what other workers doing the same task did.