Summary
Ngyuen et al’s paper discusses the rise of misinformation and the need to combat it via tools that can verify claims while also maintaining users’ trust of the tool. They designed an algorithm that finds sources that are similar to a given claim to determine whether or not the claim is accurate. They also weight the sources based on esteem. They then ran 3 studies (with over 100 participants in each) where users could interact with the tool and change settings (such as source weighting) in order to evaluate their design. The first study found that the participants trusted the system too much – when it was wrong, they tended to be inaccurate, and when it was right, they were more typically correct. The second study allowed participants to change the inputs and inject their own expertise into the scenario. This study found that the sliders did not significantly impact performance. The third study focused on gamification of the interface, and found no significant difference.
Personal Reflection
I enjoyed this paper from a 50,000 foot perspective, as they tested many different interaction types and found what could be considered negative results. I think papers that show that all work is not necessarily good have a certain amount of extra relevance – they certainly show that there’s more at work than just novelty.
I especially appreciated the study on the effectiveness of gamification. Often, the prevailing theory is that gamification increases user engagement and increases the tools’ effectiveness. While the paper is not conclusive that gamification cannot do this, it certainly lends credence to the thought that gamification is not a cure-all.
However, I took some slight issue with their AI design. Particularly, the AI determined that the phrase “Tiger Woods” indicated a supportive position. While their stance was that AIs are flawed (true), I felt that this error was quite a bit more than we can expect from normal AIs, especially ones that are being tweaked to avoid these scenarios. I would have liked to see experiment 2 and 3 improved with a better AI, as it does not seem like they cross-compared studies anyway.
Questions
- Does the interface design including a slider to adjust source reputations and user agreement on the fly seem like a good idea? Why or why not?
- What do you think about the attention check and its apparent failure to accurately check? Should they have removed the participants with incorrect answers to this check?
- Should the study have included a pre-test to determine how the participants’ world view may have affected the likelihood of them agreeing with certain claims? I.E., should they have checked to see if the participants were impartial, or tended to agree with a certain world view? Why or why not?
- What benefit do you think the third study brought to the paper? Was gamification proved to be ineffectual, or is it a design tool that sometimes doesn’t work?
Hi,
Regarding your comment about Tiger Woods, I agree that the name by itself should not be regarded as a positive token. But maybe it could if a few more words were added to form a phrase. I think this is in line with your suggestion of improved AI. Yes, I think the study would have benefited more if the AI could judge with more semantic and contextual meanings behind words.
Regarding your first question, I did not think it was a necessarily good idea. There is no assuring that the user’s input is correct, and giving the ability to adjust model weight seems a bit too much power on the user’s hand. The user could enforce his or her bias to the model and keep on dragging on the sliders until the results change.