Summary
This paper introduced a mix-initiative model which allow human and machine to check the authenticity of a claim cooperatively. This fact-checking model also considers the existing interface design principle, which supports understandability and actionability of interface. The main objective of their design is to provide transparency, supports for integrating user knowledge, and explanation of system uncertainty. To prioritize transparency over raw predictive performance, the author used a more transparent prediction model, linear models instead of deep neural networks. Further, users allow to change the reputations and stances of the system prediction. To evaluate how the system could help users to assess the factuality of claims, the author conducted three user studies with MTurk workers. The study results indicate that users might over-trust the system. The system prediction can help the user when the claim is predicted as correct. At the same time, it also degrades human performance when the system prediction errors due to the biases implicit in training data.
Reflection
I think the design of this approach is valuable. Because it does not blindly pursue the accuracy of prediction results, but also consider the transparency, understandability, and actionability of the interface. These attempts would improve the user experience since the user have more knowledge of how the system works and thus provide more trust. On the other hand, this might be the cause that the user may over-trust the system, as indicated in the paper’s experiment results. But still, I think the design of the approach is a nice try.
However, I don’t see the possibility that this system can help users. Although the design is very user friendly, it does not leverage human ability; instead, it just allows humans to participant in such a fact-checking process. Even though the design of the fact-checking process is reasonable and understandable for users, but the expectations from users side require too much mental work such as read a lot of information, thinking, and reasoning. This is a reasonable process, but it is too burdensome.
Moreover, based on the observation of the figures in the paper, I don’t think the system could facilitate the user in determining the authenticity of the claim, and I believe the experiment results also found this. Further, I found that the accuracy of the user’s judgment depends more on the type of claim. Different claims have a significant difference accuracy; this impact is even higher than the effect of the system.
It is also interesting to see the user’s feedback after they complete the task. It seems one of the users has the same opinion as me regarding the amount of information needed to read. The most impressive feedback is that the user would be confused if they have more options. I think these conditions only happen when they not sure about the correctness, and they have the right to change the system output. Finally, we can also see from the comment that when the system has the same judgment as users, users will be more sure of their answer. Still, if the system predicts results indicate the different judgments with users, this will seriously affect the accuracy of the user’s judgment. This is understandable because when someone questions your decision, no matter how confident you are, you will waver a little, let alone a machine that has 70% accuracy.
Questions:
- Do you think the system could really help humans to detect the factuality of claims? Why or why not?
- When the author design the model, to achieve the goal of transparency, they give up the higher accuracy prediction model instead of using linear models. What do you think of this? Which one is more important for your design? Transparency? Accuracy?
- What do you think of the design interface? Does it provide too much information to users? Do you like the design or not.
Word Count: 638
