Summary
The authors try to assess the accessibility of sidewalks by hiring AMT workers to analyze Google Street View images. Traditionally, sidewalk assessment is conducted in person via street audits which are highly labor intensive and expensive or by reporting calls from citizens. The authors propose using their system as an alternative for a proactive solution to this issue. They perform two studies:
- A feasibility study (Study 1): examines the feasibility of the labeling task with six dedicated labelers including three wheelchair users
- A crowdsourcing study (Study 2): investigates the comparative performance of turkers
In study 1, since labeling sidewalk accessibility problems is subjective and potentially ambiguous, the authors investigate the viability of labeling across two groups:
- Three members of the research team
- Three wheelchair users – accessibility experts
They use the results of study 1 to provide ground truth labels to evaluate crowdworkers performance and to get a baseline understanding of what labeling this dataset looks like. In study 2, the authors investigate the potential of using crowd workers to perform the labeling task. They evaluate their performance on two levels of labeling accuracy:
- Image level: tests for the presence or absence of the correct label in an image
- Pixel level: examines the pixel level accuracies of the provided labels
They show that AMT workers are capable of finding accessibility problems with an accuracy of 80.6 % and determining the correct problem type with an accuracy of 78.3%. They get better results when using majority voting as a labeling technique 86.9% and 83.9% respectively. They collected 13,379 labels, 19,189verification labels from 402 workers. Their findings suggest that crowdsourcing both the labeling task and the verification task leads to a better quality result.
Reflection
The authors have selected experts in the paper as wheelchair users, when in real life they’re civil engineers. I wonder how that would have changed their labels/results. Since accessibility in the street is not only for wheelchair users. It’s worth investigating by using a pool of multiple experts.
I also think that selecting the dataset of photos to work on was a requirement for this labeling system, else it would have been tedious amount of work on “bad” images. I can’t imagine how this would be a scalable system on google street view as a whole. The dataset requires refinement to be able to label.
In addition, the focal point of the camera was not considered and reduces the scalability of the project. Even though the authors suggest a solution of installing a camera angled towards sidewalks, until that is implemented, I don’t see how this model could work well in the real world (not a controlled experiment).
Discussion
- What are improvements that the authors could have done to their analysis?
- How would their labeling system work for random Google street view photos?
- How would the focal point of the GSV camera affect the labeling?
- If cameras were angled towards sidewalks, and we were able to get a huge amount of photos for analysis, what would be a good way to implement this project?
The different focal points and the variability of the Google Street View photos have the potential to draw the users’ attention to different points than they normally would. The experts of the test scenario would still be able to relatively easily find the crucial points. The normal users (crowd-workers) may experience the distraction and not see some of the other hard to find crucial places. To follow up on the better angle for cameras, if they were at the corners of buildings (ex. on Newman Library or similar to Burass Hall) then they would have a better spread of availability. The huge amount of publicly accessible photos and the number of cameras needed is the main issue. If there were some way to adopt some similar sensation like Pokemon Go with the cameras necessary then it may work, otherwise, normal people would not like the increase in the already abundance of cameras.