Paper:
Diakopoulos, N., Naaman, M., & Kivran-Swaine, F. (2010). Diamonds in the rough: Social media visual analytics for journalistic inquiry (Links to an external site.). In 2010 IEEE Symposium on Visual Analytics Science and Technology (pp. 115–122).
Discussion Leader: Lee Lisle
Summary:
Journalists are increasingly using social media as a way to gauge response to various events. In this paper, Diakopoulos et al. create a social media analytics tool for journalists to quickly go through large amounts of social media output that reference a given event.
In their tool, Vox Civitas, a journalist can input social media data for the program to process. First, each tweet is processed along 4 different metrics: relevance, uniqueness, sentiment, and keyword extraction. Relevance weeds out the tweets that are too delayed in the reaction to the event. If a tweet reacts to a part of the event that is fairly old, it is weeded out as it is not an initial reaction to that part. This tool is trying to assess the messages of the tweets as the event happens. Uniqueness weeds out the messages that are not unique; this mostly accounts for responses that aren’t actually stating any additional reaction. This metric also weeds out tweets that are too unique; these tweets are considered not about the actual event. The third metric, sentiment, is measured via sentiment analysis. Every tweet is processed as positive or negative to what is happening in the event. Lastly, keyword extraction pulls out popular words in the tweets that have relevance to the event. This is measured via their tf-idf score.
After explaining how their program processes the social media posts, the authors then performed an exploratory user study on their program. They found 18 participants with varying levels of experience in journalism: 7 professional journalists, 5 student journalists, 1 citizen journalist, 2 with journalism experience, and 3 untrained participants. Each of these people used the tool online remotely and answered an open-ended questionnaire.
The questionnaire had the participants run through example uses of Vox Civitas. Through the questions, the authors identified 2 primary use cases for the program: a way to find people to interview and an ideation tool. In other words, the tool could sort through the social media posts for the users so that they could find insightful or relevant people to interview. Also, the tool could help the users figure out what to write about. The questionnaire also identified how the tool would shape the angles the users would take on the social media output. Vox Civitas would help drive articles on event content or articles on audience responses to the event. Another minor angle the authors found that the participants would use is to create meta-analyses of audience response, where the participants would identify demographics of the social media post writers.
Lastly, the authors discuss ways their tool would assist with the journalistic creativity process. They state that their tool should allow journalists to skip over the initial phases of sensemaking in order for them to more quickly jump to ideation and hypothesis generation. Since the tool already processes and highlights different types of responses and shows that aggregated information via graphs and other visualizations, the journalists do not have to waste time sifting through all the data to understand it.
Reflection:
I found this paper to be a unique and in-depth user study of their program. Furthermore, I found their program to be a way for journalists to quickly understand the reaction of the crowd to an external event. This is in contrast to many of the ways we have looked at interacting with the crowd so far as this is looking at what the crowd creates or does when they are not prompted to do anything.
There were, however, a few issues I had with the paper. First, the authors acknowledge that their sentiment analysis algorithm only has a accuracy of 62.4%. They do point out that this isn’t good enough for journalists to reliably count on when looking at that data, however I would have liked to have seen them explore ways of figuring out a confidence value for the analysis or some other way of weeding out the data. As a corollary, this could have informed design of the user interface; the neutral label on the sentiment analysis visualization only meant that there weren’t posts to analyze. I felt that this would be better suited to the program showing that it couldn’t determine the sentiment of the posts.
Another issue with the paper was that it would introduce scores or rating systems without explaining them. For example, inn section 4.4 the authors mention a tf-idf score without explaining what that scoring system measures. If the authors had explained that a little better I think I could have understood their methodology for extracting keywords significantly more.
I appreciated the focus the authors placed on their user interface; breaking down what each part measured or what it conveyed was helpful to understand the workflow. Furthermore, their statistics in section 6.4.3 that detailed how much each part of the interface was used was a good way to illustrate how useful each part of the tool was to the participants. It also conveyed that the participants did take advantage of the features the authors supplied and they were able to understand their usage.
Questions:
- Do you think this tool could be used in fields other than journalism? How would you use it?
- The authors used a non-lab study to enhance the ecological and external validity of the study, and tracked how the users interacted with the interface. Do you think any data was lost in this translation?
- The professional journalists were noted to have not used the keyword functionality of the interface, and the authors did not follow up with them to find out why. Do you have any idea why they might have avoided this functionality?
- The participants noted that one way of using this tool was to figure out any links between demographics of the audience and their responses. Have you seen this in more recent media?
