1. Mitra, G. P. Wright, and E. Gilbert, “A Parsimonious Language Model of Social Media Credibility Across Disparate Events”

Written language for centuries, coming from word of mouth, has been the primary mode for discourse and the transportation of ideas. Due to its structure and capacity it has shaped our view of the world.But, due to changing social landscapes, written language’s efficacy is being tested. The emergence of social media, preference of short blobs of text, citizen journalism, the emergence of cable reality show, I mean, the NEWS and various other related occurrences are driving a change in the way we are informed of our surroundings. These are not only affecting our ability to quantify credibility but is also inundating us with more information one can wade through. In this paper, the author explores the idea of whether language from one of the social media website, twitter, can be a good indicator of the perceived credibility of the text written.

The author tries to predict the credibility of the credibility of news by creating a parsimonious model(low number of input parameter count) using penalized ordinal regression with scores “low”, “medium”, “high” and “perfect.” The author uses CREDBANK corpus along with other linguistic repositories and tools to build the model. The author picks modality, subjectivity, hedges, evidentiality, negation, exclusion and conjugation, anxiety, positive and negative emotions, boosters and capitalization, quotations, questions and hashtags as its linguistic features while using number of number of original tweets, retweets and replies, average length of original tweets, retweets and replies and number of words in original tweets, retweets and replies as the control variables. Measures were also taken like the use of penalized version of ordered logistic regression to handle multicollinearity and sparsity issues. The author then goes on to rank and compare the different input variables listed above by its explanatory powers.

One of the things I was unsure of after reading the paper is if the author accounted for long tweets where the author uses replies as a mean to extend one’s tweet. Eliminating this could make the use of number of replies as a feature more credible. One could also see that, the author has missed to accommodate for spelling mistakes and so forth, as this preprocessing step could improve the performance and reliability of the model.
It would be an interesting idea to test if the method the author describes can be translated to other languages especially languages which are linguistically different.
Language has been evolving ever since its inception. New slangs and dialects adds to this evolution. Certain social struggles and changes also have an impact on language use and vice versa. Given such a setting, is understanding credibility from language use a reliable method? This would be an interesting project to take on to see if these underlying lingual features have remained same across time. One could pick out texts involving discourse from the past and see how the reliability of the model build by the author changes if it does. But this method will need to account for the data imbalance.
When a certain behaviour is penalized, the repressed always find a way back. This can also be applicable to the purveyors of fake news. They could game the system in using certain language constructs and words to evade the system. Due to the way the system is build by the author, it could be susceptible to such acts. In order to avoid such methods one could automate this feature selection. The model could routinely recalculate the importance of certain features while also adding new words into its dictionary.
Can a deep learning mode be built to better the performance of credibility measurement? One could also try building a sequential model may it be LSTMs or even better a TCN [2] to which vectors of words in a tweet generated using word2vec could be given as input along with some attention mechanism or even [4] to allow us to have an interpretable model. Care has to given that models especially in this area have to be interpretable model so as to avoid not having an accountability in the system.

[2] Colin Lea et al, “Temporal Convolutional Networks for Action Segmentation and Detection”
[3] T. Mikolov et al, “Distributed Representations of Words and Phrases and their Compositionality”
[4] Tian Guo et al, “An interpretable {LSTM} neural network for autoregressive exogenous model”

Read More

1. Mitra, G. P. Wright, and E. Gilbert, “A Parsimonious Language Model of Social Media Credibility Across Disparate Events”

SUMMARY:

This paper proposes a model that aims in classifying the credibility level of a post/tweet as one of Low, Medium, High or Perfect. This is based on 15 linguistic measures including modality, subjectivity which are lexicon-based measures and questions, hashtags that are not lexicon based. The study uses a CREDBANK corpus which contains events, tweets and crowdsourced credibility annotations. It not only takes into consideration the original tweet but also retweets and replies to the original tweet and other parameters such as tweet length. The penalized ordinal regression model shows that several linguistic factors have an effect on perceived credibility most of all subjectivity followed by positive and negative emotions.

REFLECTION:

1.  The first thing that I was concerned about was tweet length, this was set as a control. We have, however, in the past, discussed as to how shorter tweet lengths tend to be perceived as truthful because the tweeter wouldn’t have much time to type in a tweet while in the middle of a major event. The original tweet length itself negatively correlated with perceived credibility.
2. The language itself is constantly evolving, wouldn’t we have to continuously train with newer lexicon as time goes by? 10 years ago, the word ‘dope’ and ‘swag’ (nowadays used interchangeably with amazing or wonderful) would have meant some very different things.
3. A well-known source is one of the most credible ways of getting news offline. Perhaps combining the model with one that tests perceived credibility based on source could give us even better results. Twitter has some select verified accounts that have higher credibility than others. The platform could look to assign something akin to karma points for accounts that have in the past given out only credible information.
4. This paper has clearly outlined that some words evoke the sense of a certain tweet being credible more than some others. Could these words be intentionally used by miscreants to seem credible and spread false information? Since this model is lexicon based, it is possible that the model cannot automatically adjust for it.
5. One observation that initially irked me in this study was that the negative emotion was tied to low credibility. This seems about correct when we think about how the Kubler-Ross model’s first step is denial. If this is the case, I first wondered how anyone was going to be able to deliver bad news to the world ever. However, taking a closer look at the words that have a negative correlation specifically are ones that seem negatively accusatory (cheat, distrust, egotist) as against sad (missed, heartbroken, sobbed, devastate). While we may be able to get the word out about say a tsunami and be believed, outing someone to be a cheat may be a little more difficult.

Read More

1. Garrett, R.K. and Weeks, B.E., 2013, February. The promise and peril of real-time corrections to political misperceptions. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 1047-1058). ACM.
2. Mitra, T., Wright, G.P. and Gilbert, E., 2017, February. A parsimonious language model of social media credibility across disparate events. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 126-145). ACM.

Summary and Reflection for paper 1

Article one talks about how people tend to be picky and choosy when it comes to rumors and their correction. They find a news hard to believe if it doesn’t align with their preconceived notions about an idea, and even harder to made amends for proliferation of a false news if it does align with their agenda/belief.  It presents plausible recommendations about fine graining the correction into users view so that it is more easily digestible and acceptable. I personally related with recommendation 2 about letting the user know about the risks associated with hanging on to the rumor, or the moral obligation of correcting their views.  However, does the same user profiling and algorithms for guessing preferences work across sources of news other than the traditional ones i.e. twitter, CNN etc.?

As delayed correction seemed to work better in most of the cases, can a system decide how likely the user is to pass on the news to other sources based on his/her profile, present real-time corrections to users who tend to proliferate fake news faster than others by using a mix of all three recommendations presented in this paper?

 

Summary for paper 2

As long as there’s market for juicy gossips and misinterpretation of events, rumors will keep spreading in one form or the other. People have a tendency to anticipate, and readily believe things which are either consistent with their existing beliefs, or give an adrenaline rush without potentially bringing any harm to them.  Article 2 talks about using language markers and cues to authenticate a news or its source which can, when subsumed with other approaches of classifying credibility, work as an early detector of false news.

Reflection and Questions

• Credibility score can be maintained and publicly displayed for each user, which starts from 0 and is decreased every time the user is reported for posting or spreading a misleading news .Can such credibility score be used to determine how factual someone’s tweets/posts are?
• Can such a score be maintained for news too?
• Can a more generous language model be developed, which also takes multilingual postings into account?
• How can number of words used in a tweet, retweet and replies be an indicator of authenticity of a news?
• Sometimes users use emoticons/emojis in the end of the tweet to indicate satire and mockery of the otherwise seriously portrayed news. Does the model include their effect on the authenticity of the news?
• What about rumors posted via images?
• So much propaganda is spread via videos or edited images on the social media. Sometimes, all textual news that follows is the outcome of a viral video or picture circulating around the internet. What mechanism can be developed to stop such false news from being rapidly spread and shared?

Read More

Today’s topic of discussion is Credibility and Misinformation online.

Mitra, Tanushree et al. (2017) – “A Parsimonious Language Model of Social Media Credibility Across Disparate Events”- CSCW 2017 (126-145).

Summary

The paper mainly focuses on establishing the credibility of news across social media. The authors identified 15 theoretically grounded linguistic assumptions and took help of the CREDBANK corpus to construct a model that would map language to the perceived levels of credibility. Credibility has been broadly described as believability, trust and reliability along with other related topics. However, the term credibility has been termed as both subjective or objective depending on the area of expertise of the researcher. A CREDBANK [1] was constructed which is essentially a corpus of tweets, topics, events and associated human credibility judgements. The corpus has credibility annotations on a 5-point scale (-2 to +2). The paper dealt with the perceived credibility (annotations based as “Certainly Accurate”) of the reported twitter news of a particular event. Proportions of annotations (P_ca = “Certainly Accurate” ratings of event / Total rating for that event) was calculated. An event was rated as “Certainly Accurate” if its P_ca belonged to the “Perfect Credibility class” (0.9≤ P_ca ≤1). All events were given a credibility class of Low to Perfect (rank as Low ≤ Medium ≤ High ≤ Perfect). The linguistic assumptions were considered as the potential predictors of perceived credibility. The potential credibility markers were namely, Modality, Subjectivity, Hedges, Evidentiality, Negations, Exclusions and Conjugations, Anxiety, Positive and negative emotions, Boosters and Capitalization, Quotation, Questions and Hashtags. Nine variables were used as controls namely, Number, average length and number of words in original tweets, retweets and replies. The regression technique used an alpha (=1) parameter to determine the distribution of weight amongst the variables. It was found out that retweets and replies with longer message lengths were associated with higher credibility scores whereas, higher number of retweets were correlated with lower credibility scores.

Reflection

It has become increasingly common for people to experience news through social media and with this comes the problem of the authenticity of that news. The paper dealt with few credibility markers which assessed the credibility of the particular post. It spoke about the variety of words used in the post and how they are perceived to be.

Firstly, I would like to point out that certain people have their own jargon. The millennials speak in a specific language, a medical professional may use a certain language. This may be perceived as negative or dubious language which may in turn reduce the credibility.  Does the corpus have variety of informal terms and languages as well as group specific languages in the database to avoid erroneous result?

Additionally, a statement in the paper says, “Moments of uncertainty are often marked with statements containing negative valence expressions.” However, negative expressions are also used to depict some unfortunate event. Let’s take the example of the missing plane MH 370. People are likely to use negative emotion while tweeting about that incident. This certainly doesn’t make it uncertain or less credible.

Although this paper dealt with the credibility of news in the social media realm, namely twitter, credibility of news is still a valid concern when it comes to all forms of news sources. Can we apply this to Television and Print media as well? They are often accused of reporting unauthenticated news or even being bias in some cases. If a credibility score of such media is also measured other than the infamous “TRP or Rating”, it would make these news outlets credible as well. It would force the news agencies to validate their source and this index or score would also help the readers or followers of the network to judge the authenticity of the news being delivered.

[1] Mitra, Tanushree et.al. (2015)- “CREDBANK: A Large-scale Social Media Corpus With Associated Credibility Annotations“

Read More

In [1], the work presented has a very strong argument for the need for language models in social computing platforms. This can be deconstructed using the following  two sections:

• SUMMARY: The paper first gives a theoretical base to the concepts that are used, along with a survey of related work. Here modality, subjectivity and the other linguistic measures used have been defined to capture the different perceived dimensions of a language model. The claims of all of them are warranted with the help of previous work. The statistical framework considered the problem as an ordered logistic regression one resulting in phrase collinearity (A common property in natural language expressions). The performance of the model is well documented with a sound defense for the validity. The overall accuracy of the model is a clear indicator of its use against the considered baseline classifiers. Implications are drawn on each of the defined measures based on the inferential statistical results of the model.
• REFLECTION: As a proponent for credibility level assessments of social media content, I favor the establishment of well-founded metrics to filter content. The paper is a strong step in the direction with a detailed account of the design process for a good linguistic model. The few immediate design opportunities that are regurgitated from the paper are:
  • The creation of a deployable automated system for content analysis adopting such a model. This can be a very interesting project where a Multi-agent Machine learning model using the CREDBANK system as its Supervised Learner, can help classify tweets in real time assigning credits to the source of the content. This will be monitored by another agent which reinforces the Supervised Learner, essentially creating a meta-learner.[2]-[5]
  • Adaptation of an ensemble of such models to form a global system which cross-verifies and credits information not just from a single platform but across multiple ones to give the metaphorical “global mean” as against the “local mean” in information. [6]
  • The model should account for the linguistic chaos even with newly created “Purrrrr” or “covfefe”. These lexiconic outliers could be captured with the use of Chaos Theory in Reinforcement Learning, which could be an entirely new avenue of research.

The paper also helped me understand the importance of capturing the different dimensions of a language model and corroborating it with evidence with tools of statistical inference.

[1]        T. Mitra, G. P. Wright, and E. Gilbert, “A Parsimonious Language Model of Social Media Credibility Across Disparate Events,” Proc. 2017 ACM Conf. Comput. Support. Coop. Work Soc. Comput. – CSCW ’17, pp. 126–145, 2017.

[2]        D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Learning to Generalize: Meta-Learning for Domain Generalization,” Oct. 2017.

[3]        F. Sung, L. Zhang, T. Xiang, T. Hospedales, and Y. Yang, “Learning to Learn: Meta-Critic Networks for Sample Efficient Learning,” Jun. 2017.

[4]        R. Houthooft et al., “Evolved Policy Gradients,” Feb. 2018.

[5]        Z. Xu, H. van Hasselt, and D. Silver, “Meta-Gradient Reinforcement Learning,” May 2018.

[6]        Marios Michailidis (2017), StackNet, StackNet Meta-Modelling Framework, URL https://github.com/kaz-Anova/StackNet

Read More

Mitra, Tanushree, Graham P. Wright, and Eric Gilbert. “A parsimonious language model of social media credibility across disparate events.” Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 2017.

This paper represents an in-depth and meticulous analysis of different linguistic and social network features that affect the credibility of a post. Mitra et. al. take intuition from linguistic cues such as subjectivity, positive emotions, hedging along with social network specifics such as retweets and replies to build a model that maps such features to the level of credibility. This study with thorough experimentation and validation not only provides strong evidence of the effects of such features but also gives qualitative insights and implications of the research.

The model fit comparison section specifically reflects several social network characteristics. For example, getting better explanatory power after including both the original texts and replies highlights the role of context and conversational nature of the social media interaction. Seeing the low predictive power of non lexicon based features, such as hashtags, caps and question marks, I am curious about whether all such features could be grouped into the “readability index” of the corpus corresponding to each event. It is possible that lower readability can be a good predictor of lower credibility. (Although, it is not clear just by intuition whether higher readability will be a good predictor of higher credibility)

Credibility in non-anonymous networks can have strong ties to how the source is viewed by the reader. Authors discuss that they did not include source credibility in the features but I think that “poster status” can also impact the perceived credibility. For example, I am more likely to believe in the fake news posted by my colleagues rather than a stranger with the same source. Similarly, I am more likely to believe in the information provided by a user with higher karma points than one with the lower karma points. Because the credibility annotations were done by turkers, it is not possible to assess the effect of poster status in the current setup. But, in a retrospective study, it is possible to have additional non-lexicon based features such as user statistics and tie strengths between the poster and the reader.

Such analysis that comprises of strong linguistic and non-linguistic features can be also applied to detecting fake news. Websites such as “Snopes”, “PolitiFact”  have pieces of news and the fact-check review on them tagged by “original content”, “fact rating” and “sources” which can be used either for stand-alone analysis or grouping the twitter event streams as fake or credible.

Finally, I believe that consequences of credibility range from disbelieving in scientific and logical information such as the importance of vaccinations and climate change to believing in conspiracy theories and propaganda.  Fast paced online interactions do not allow the users to analyze every piece of information they get. This makes the linguistic and social influence perspective on credibility more relevant and important in de-biasing the online interaction.

Read More

A Parsimonious Language Model of Social Media Credibility Across Disparate Events

Summary:

In this paper, the authors uncover that language used in tweets can indicate whether an event will be perceived as highly credible or less credible. After examining millions of tweets corresponding to thousands of twitter events they identify 15 linguistic features that can act as predictors of credibility and present a parsimonious model that maps linguistic constructs to different levels of credibility (“Low” < ”Medium” < ”High” < ”Perfect”). The events were annotated based on the proportion of annotations rating the reportage as ‘Certainly Accurate’. Authors use Penalised Logistic Regression for modeling and find that subjectivity is the most explanatory feature followed by positive & negative emotion. Overall, the results show that certain words and phrases are strong predictors of credibility.

Reflection:

As defined by Hiemstra et al in their paper “Parsimonious Language Models for Information Retrieval”, a parsimonious model optimizes its ability to predict language use but minimizes the total number of parameters needed to model the data.

All the papers we read for the last two reflections stressed the fact how linguistic constructs define the identity of a group/individual on online social communities. While the he use of racist, homophobic and sexist language was part of the identity of /b/ board in 4chan, users of a group in Usenet used “Geek Code” to proclaim their geek identity. We also learned how banned users on CNN used more negative emotion words and less conciliatory language.

I liked how authors validated their method of annotating the events with the HAC based clustering approach to group the events. They use Rand similarity coefficient to find the similarity between the two clustering techniques. The high R value indicates agreement between the two. I agree with author’s selection of annotation technique since it’s more generalizable.

Each mechanical turk needs to be aware of the event before annotating it. Otherwise they need to search for it online. How can can we ensure that the online news is not making the turker biased. Are turkers reading all the tweets in the pop up window before selecting a category or do they just base their decision by reading the first few tweets. I believe how an event is reported can greatly vary. So making a judgment by reading the first few tweets might not give a clear picture. Also, was the order of tweets in pop up window same for all the turkers? I believe I’ll find the answers to these questions after reading the Credbank paper.

The unit of analysis in this paper is an event rather than a tweet. And an event is perceived highly credible if large number of annotators rate the reportage as ‘certainly accurate’. But is the news perceived as credible actually credible? It will be interesting to see whether events perceived as credible are actually credible or not. A lot of work is going on in fake news detection and rumor propagation on social media platforms. Also, can people/organizations make use of this research to structure rumors in such a way that they are perceived credible? Will this reverse approach work too?

I believe a better way of calculating the value of “Questions” feature would be to calculate the proportion of tweets carrying question mark rather than counting the total number of question marks present in the tweets corresponding to an event.

One of the other features to determine credibility could be presence of URL in tweets. Specially URLs of trusted news agencies like CNN.

In the end I’ll reiterate the author and say that linguistic features combined with other domain specific features could act as foundation for an automated system to detect fake news.

Read More

Reading:

[1] A Parsimonious Language Model of Social Media Credibility Across Disparate Events

Summary:

This paper is geared towards understanding perceived credibility of information that is disseminated using social media platforms. In order to understand perceived credibility of published information, the authors of this paper decided to examine 66 million twitter messages a.k.a tweets, which were associated with 1,377 events that occurred over a period of 3 months — between October 2014 and February 2015. In order to examine these tweets from a linguistic vantage point, the authors came up with fifteen linguistic dimensions that assisted them to conceive a model that “maps language cues to perceived credibility.” In addition to the various linguistic dimensions the authors also highlighted the importance of particular phrases within these dimensions. To establish credibility of various tweets, the authors executed various experiments where the subjects were asked to rate a tweet on a 5 point Likert scale ranging from -2 (certainly inaccurate) to +2 (certainly accurate). The authors also employed nine control variables — in addition to the results from the experiment, linguistic dimensions, and identification of various phrases within these dimensions — that helped them account for the effect of  content popularity. The culmination of myriad of linguistic and statistical models ensued in a definitive parsimonious language model — howbeit the authors warn against independent usage of this model. Albeit they argue that the language model serves as an important step towards a fully autonomous system.

Reflection and Questions:

Comprehending the fact that utilization of specific words, writing styles, and sentence formations can alter the perceived credibility of a post made on social media surprises me, partly because its a new finding for me and partly because as an engineer I have never really paid attention to language formation but only to the facts in the text. Throughout the paper the authors have engaged in the idea of credibility of the post/tweet, howbeit according to my understanding it is the source of the information that necessitates “credibility” and text presented by the source necessitates “accuracy” and  “reliability”.  The authors write that “words indicating positive emotion were correlated with higher perceived credibility;” the question then arises: what about news bearing bad news? for instance death of a world leader; that news will not bear any “positive emotion.” Whilst reading the paper I came across a sentence stating that disbelief elicits ambiguity, which I disagree with. Disbelief can be used in a variety of combinations, none of which I think elicit ambiguity.

Reading the paper, I couldn’t help but think how does this model utilize slang language ? There could be a credible post that involves slang language because according to me millennial’s are more prone to trust a post that contains colloquial language instead of formal language, unless the source of information is associated with main stream media. The previous question alludes to the next question how is slang language in different countries/ usage of English in other countries taken into consideration ?  The reason I ask this is because specific words have different interpretations in different countries/ different regions of the same country resulting in different perceived credibility. As we are on the topic of interpretation of language in different regions, the question arises that: is this model universally suitable for all the languages in the world (with slight alterations), or would different languages require different models ? The main reason for this question is because people tweet in varied languages and language barrier could change perceived credibility of the post/tweet. Lets hypothesize that a post is originally made in a language other than English, howbeit English readers use the translate button on twitter/facebook to read the post, now the perceived credibility of post depends on the region the person resides in and the accuracy of translate feature of the particular social media platform. How can multiple considerations be synthesized to create a more suitable perceived credibility score for a specific situation ? 

 

Read More

Mitra, Tanushree, Graham P. Wright, and Eric Gilbert. “A parsimonious language model of social media credibility across disparate events.” Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 2017.

 

Summary:

The authors of this paper examined how people perceived the credibility of events that were reported by users. The goal was to build a model that would be able to identify, with a good degree of accuracy, how credible a person will perceive an event based on the text of the tweets (and replies to the tweets) related to the event. To do this, they used a dataset of collected twitter streams with the tweets classified by events and the time when they were collected. These tweets were also rated for their credibility on a 5 point scale. The authors also put forward 15 language features that could be used to influence the credibility perception. With all this in hand, the authors were able to analyze and identify the words and phrases in the different language feature categories that corresponded to high and low perceived credibility scores for the various events. Though the authors advise against using the model on its own, it can be used to complement other systems such as fact checking systems.

 

Reflection:

What I found most interesting was the phenomenon of how the perception of credibility seems to flip for positive to negative and vice versa between the original tweets and replies. I first thought there might be a parallel here between tweets-replies and news article-comments but of course that doesn’t work because there are cases where the replies are perceived more credible than the original so that parallel doesn’t always work because the original tweets are not always credible. (Then again, there are cases where a news article is not necessarily credible so maybe there is a parallel here after all? I’m sorry, I’m grasping at straws here.)

“Everything you read on the internet is true. – Mahatma Gandhi.” This is a joke that you’ll sometimes see on Reddit but also serves as a warning against believing everything you read because you perceive it to be credible. The authors of this paper mentioned how the model can be used to identify content with low credibility and boot it from the platform before it can spread. But could it also be used by trolls or people with malicious intent to augment their own tweets (or other output) in order to increase the perceived credibility of their tweets? This could certainly cause some damage as we are talking about false information being more widely believed because it was improved thanks to algorithmic help where otherwise it may have had a low perceived credibility.

Another thing to consider is longer form content. This analysis is necessarily limited by their dataset which only consists of tweets. But I have often found that I am more likely to believe something if it is longer and articulate. This is especially apparent to me when browsing Reddit where I am more likely to believe a well written, long paragraph or a multi paragraph comment. I try to be skeptical but I still catch myself believing something because it happens to be longer (and also believe it more when it is gilded, but that’s for a different reflection). So the question that arises is: What effect does length have on perceived credibility? And how do the 15 language factors the authors identified affect perceived credibility in such longer form text?

 

Read More