1. Bond, Robert M., et al. “A 61-million-person experiment in social influence and political mobilization.” Nature 489.7415 (2012): 295.
2. Kramer, Adam DI, Jamie E. Guillory, and Jeffrey T. Hancock. “Experimental evidence of massive-scale emotional contagion through social networks.” Proceedings of the National Academy of Sciences 111.24 (2014): 8788-8790.

Both the papers provide interesting analysis of user generated data on Facebook. As far as I remember, the key idea behind the first paper was briefly discussed in one of the early lectures. While, there might be some ethical concerns regarding the data collection, usage and human subject consent in both the studies, I find the papers to be very relevant and thought provoking in today’s world where social media is more or less an indispensable part of everyone’s lives. The first paper by Bond et. al. discusses a randomized controlled trial of political mobilization messages on 61 million Facebook users during the 2010 U.S. congressional elections. The experiment showed an ‘informational’ or ‘social’ message at the top of the news feed of Facebook users in the U.S. (18 years of age and above) as shown in the image below.

Approximately 60 million users where shown the social message, 600 K users were shown the social message and 600 K users were not shown any message adding up to the ‘61-million-person’ sample advertised in the title. The key finding of this experiment was that messages on social media directly influenced political self-expression, information seeking and real-world voting behavior of people (at least those on Facebook). Additionally, ‘close-friends’ in a social network (a.k.a. strong ties) are responsible for the transmission of self-expression, information seeking and real-world voting behavior. In essence, strong ties play a more significant role in spreading online and real-world behavior as compared to ‘weak ties’, in online social networks. Next, I summarize my thoughts on this article.

The authors find that users who received the social message (instead of the plain informational message) where 2.08% more likely to click on the ‘I Voted’ button. This seems to suggest a causality between the presence of images of friends who pushed the ‘I Voted’ button and the user’s decision to push the ‘I Voted’ button. I am not convinced with this suggestion because of the huge difference in the sample size of the social and informational message groups. I believe online social networks are complex systems and spread of behaviors (contagions) in such systems is a non-linear and emergent phenomenon. I feel that ignoring the differences between the two samples (in terms of network size and structure) is a little unreasonable while making such comparisons at the gross level. I feel this particular result will be more convincing if the two samples were relatively similar and the findings were consistent for repeated experiments. Another interesting analysis could be to look at, which demographic segments are influenced more by the social messages as compared to the informational messages. Is the effect, reversed for certain segments of the user population? Lastly, approximately 12.8% of the 2.1 billion user accounts on Facebook are either fake or duplicate. It would be interesting to see how these accounts would affect the results published in this article.

The second article by Kramer et. al. suggests that emotions can spread similar to contagions, from one user to another in online social networks. The article presents an experiment wherein the amount of positive and negative posts in the News Feed of Facebook users was artificially reduced by 10%. The key observation was that, when positive posts were reduced the amount of positive words in the affected user’s status updates decreased. Similarly, when negative posts were reduced the amount of negative words in the affected user’s status updates decreased. I think this result suggests that people innately reciprocate the emotions they experience (even in the absence of nonverbal cues) acting like feedback loops. I feel that the weeklong study described in the article is somewhat insufficient to support the results. It might also be more convincing if the experiment was repeated and the observations remained consistent each time. Another thing that I feel is missing in the article is statistics about the affected users status updates, i.e. what was the mean, std. dev. of the number status updates posted by the users. Additionally, it is important to know if the users posted status updates only ‘after’ reading their News Feeds? And if this ‘temporal’ information is captured in the data at all? Based on my limited observations on Facebook status updates, I feel most of the time they relate to the daily experiences of the user. For example, visit to a restaurant, a promotion, successful defense, holidays or trips. I feel it’s very important that we avoid ‘Apophenia’ when it comes to this kind of research. Also, it is unclear to me why the authors have used Poisson regression here and what is the response variable?

Read More

Summary – 1

Reflection – 1

Firstly, it is difficult to imagine how large 61 million really is! I was personally quite awed at the scale of experimentation and data collection through online social media and what they can tell us about human behavior.

This paper dealt with multiple issues and could have been a longer paper. An immediately interesting observation is that the effect of online appeals to vote is very small. In more traditional survey-based experiments, this increase would probably not have been noticeable. However, I found it odd that the “social message” group consisted of over 60 million people, however, the “informational message” and control group had only ~600,000. The imbalance seems uncharacteristic for an experiment of this scale and I am curious what the class thought about the distribution of the dataset.

Finally, I found the definition of the close friends arbitrary. The authors define close friends as users who were in the eightieth percentile or higher of frequency of interaction. This definition seems engineered to retrofit the observation that 98% of users had at least one close friend and an average of 10 close friends.

Summary – 2

The 2014 paper on emotional contagion on social networks is mired in controversy. The researchers ask the question – do emotional states spread through social networks? Quoting verbatim from the paper:

The experiment manipulated the extent to which people (N = 689,003) were expose to emotional expressions in their News Feed.

The researchers adapt LIWC to their News Feed algorithm to filter out positive or negative content. They find that when negativity was reduced, users posted more positive content, and consequently when positivity was reduced, the users post less positive content.

Reflection – 2

While the experiment itself is within the realms of Facebook’s acceptable data use policy, there are several signs in the Editorial Expression of Concern and Correction that this experiment may not pass the Institutional Review Board. However, if these two experiments are deemed ethical, they are examples of great experiment design.

Neither paper builds any model. They rely on showing correlation between their dependent and independent variables. As with the previous paper, the effects are small. But applied to large population, even small percentage increases are large enough to take note of.

Questions

1. I personally find it quite scary neither study can be done by a university and explicitly needs access from Facebook (or any other social media) for that matter. The consolidation of social media analytics capability in the hands of a few may not bode well for ordinary citizens. How can academic research make this data more available and reachable?
2. The first paper lays basis for how societies can be influenced online. Can this be used to target only a small section of users? Can this also be used to identify groups that are under-represented and vocalize their opinions?

Read More

[1]. Niculae, V., Kumar, S., Boyd-Graber, J., & Danescu-Niculescu-Mizil, C. (2015). Linguistic harbingers of betrayal: A case study on an online strategy game. arXiv preprint arXiv:1506.04744.

This research paper explores linguistic cues in a strategy game ‘Diplomacy’ to examine patterns that foretell betrayals. In Diplomacy each player chooses a country and tries to win the game by capturing other countries. Players form alliances and break those alliances, sometimes through betrayal. The authors have tried to predict a possible betrayal based on sentiment, politeness, and linguistic cues in the conversation between players. The authors collected data from 2 online platforms that comprised of 145k messages from 249 games. The authors predict betrayal with 57% accuracy and find a few interesting things such as: betrayer has more positive sentiment before betrayal, less planning markers in betrayer’s language and have more polite behavior.

I thoroughly enjoyed reading this paper………. until section 4, where they explain modeling. I felt that either modeling process was poorly performed or otherwise should have been explained better. All they say is that “expert humans performed poorly on this task”, but what did those experts do? What does poorly mean? After that, they built a logistic model after univariate feature selection and best model achieves a cross-validation accuracy of 57% and F₁ score of 0.31. What is the best model? They did not explain their “best model” and what features went into it. Secondly, is 57% accuracy good enough? A simple coin toss would have given us 50%. They also had various findings on eventual betrayer such as: he is more polite before betrayal etc. but what about the cases when betrayal does not happen? I felt that they only explained statistics for cases with betrayals.

Finally, can we generalize the results of this research? I can claim with 100% certainty that everyone will say “NO” because human relations are much more complex than simple messages of attack or defense. I appreciate the fact that authors have addressed that they do not expect the detection of betrayal to be solvable with high accuracy. But supposing 57% significant enough, can we generalize it to real-world scenarios that are similar to Diplomacy? For example: a betrayal from your office colleague working on a same project as you but taking all the credit to gain promotion. Can we detect that kind of betrayal from linguistic cues? Can we replicate this research on similar real-life scenarios?

Read More

Summary:

The paper finds out the linguistic and discourse cues that generally occurs before the betrayal. Niculae et.al. study user behavior on popular online diplomatic games and analyze contexts, sentiment, language and discourse used by both parties (victim and betrayer) in different seasons of the game. They observe that betrayals occur generally when there arises an imbalance of sentiment, politeness and discourse marks between the parties. This study is the primary step to create a model that can forecast betrayals and help understanding friendships in diplomatic relations in real life.

Analysis:

This paper tries to find out the impact of linguistic cues before the occurrence of betrayal. For understanding this, they also compute sentiment, discourse, and politeness of the user from both parties in online behavior. I like the idea of the paper and their analysis how they show the imbalance between the communication between two parties cause betrayal by one. My question is: will this be applicable in real life also? Because online game is played mostly by inexperienced teenagers who have very little about diplomacy and complex relationships. Besides, as a future research direction, we can think of creating a model using series of events happened in historical data of diplomacy that can predict betrayals in diplomatic and political relations in real life.

Read More

Niculae, Vlad, et al. “Linguistic harbingers of betrayal: A case study on an online strategy game.” arXiv preprint arXiv:1506.04744 (2015).

This research paper explores linguistic cues in a game ‘Diplomacy’ (a strategy game) where players form alliances and break those alliances through betrayal. The authors have tried to predict a possible betrayal based on following attributes:  positive sentiment, politeness, and structured discourse. However, in my opinion there can be other factors like body language and facial expressions of a player which can also determine a possible betrayal.

The authors have collected data from 2 online platforms. The dataset comprises of 145k messages from 249 games.  Diplomacy is unique in a way that all players submit their written orders and these orders are executed simultaneously; there is no randomness. Hence the communication of the players depends only on the communication, cooperation, and movement of players.

In sec 3 of the paper, authors talk about relationships and stability and how interactions within the game define the relationship between players. The authors have used external tools for sentiment analysis and politeness classification. The authors have built a binary classifier to predict whether a player is going to betray another player.Such computations might give satisfactory results in a game scenario; however, they cannot be extended to real-life scenarios.

In the end, the paper explores relationships in a war based strategy game which doesn’t quite relate to the real world and looks quite unrealistic.

Read More

Paper: Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game

Summary:
This paper tries to find linguistic cues that can predict “betrayal” in an online game called “Diplomacy”. Diplomacy is an online war strategy with a little different order of actions. Here all the player perform their moves at the same time. This makes it very similar to the problem of “Prisoner’s dilemma”. For this reason, players make and break alliances. The main communication medium for the game is through messages. The authors collected messages between users to find out what type of language cues are used by the users before betraying alliance. The authors found that betrayers express more positive sentiment, politeness while less argumentation and planning in their messages. Based on these attributes authors create a model to predict betrayal which achieves about 57% cross-validation accuracy.

Reflection:
Authors’ choice of Diplomacy was a good source for analyzing the interaction of betrayer and victim. The authors in this paper are overly protective about the effect of time on their relationship which is surprising from the given result when they just ignore the status of the game in predicting betrayal. One of the results of the study that doesn’t make sense is that betrayer’s plan less while the victim’s do more. This is counter-intuitive in the sense if the victim plans more it is likely that they have a better grasp of different situations in the game. This begs the question: what is the effect of different level of experience in the game on betrayal? It is unlikely that a novice player (noob) will betray his alliances. Another thing noticeable in the tables in the paper is the absence of the value of the coefficient for both positive features and negative features in their prediction power. Although this paper provides some interesting insight into the behavior of a betrayer, it doesn’t seem to have any direct application in real life.

Read More

Niculae, V., Kumar, S., Boyd-Graber, J., & Danescu-Niculescu-Mizil, C. (2015). Linguistic harbingers of betrayal: A case study on an online strategy game. arXiv preprint arXiv:1506.04744.

Summary:

The paper explores linguistic cues that indicate fickle interpersonal relations, like close friends becoming enemies. Since data that define the relationship between friends or enemies are not extensively accessible, researchers turn to a war-themed strategy game in which friendships and betrayals are orchestrated primarily through language. By studying dyadic interactions in the game and analyzing languages under cases that players form alliances and betray each other, they characterize subtle signs of imminent betrayal in players’ conversation and examine temporal patterns that foretell betrayal. From conversation scenarios in Diplomacy (the war-themed strategy game), we can actually see that betrayer would unconsciously reveal their planned treachery, meanwhile, the eventual victim can rarely be able to notice these signals.
They find that if the balance of conversational attributes like positive sentiment, politeness, and structured discourse shows sudden changes, imminent betrayal would happen in the future conversation. Then researchers provide a framework for analyzing communication patterns and explore linguistic features that are predictive of whether friendships will end in betrayal. They also discuss how to generalize methods to other domain and how automatically predicting relationships between people can help advance the study of trust and relationships using computational linguistics.

Reflection:

“Despite people’s best effort to hide it, the intention to betray can leak through the language one uses.” This reminds me of another idea that we may use the same strategy to detect people’s relationship pattern. Under normal circumstances, there are different relationship patterns when people getting along with each other. For example, some are balanced relationships like friends, colleagues, and relatives, while others are unbalanced relationships like leaders and subordinates, professors and students. Through the conversation content between people, we can extract the linguistic features following the same direction such as sentiment, argumentation, and discourse, politeness, and talkativeness, to predict the possible relationship between people. By analyzing the patterns of interpersonal relationships, we can have a deeper understanding of the current status of people’s life or whether the patterns will change over time, which is a more macro-sociological issue to figure out.
Apart from detecting the relationship pattern, we may use specific semantic features to study much other potential information hidden in the conversation, such as trust, familiarity, and intimacy between people.
Since the intuition of this paper is that a stable relationship should be balanced, it makes sense that all the predictions of Betrayal in the paper is based on signal an imbalance in the communication patterns of the dyad. However, my concern is whether these mentioned semantic features could provide a complete and efficient predictive analysis. Are there other available properties? e.g., Humor, straightforwardness. Or, in addition to detecting the imbalance of both sides in conversation, we may analyze the change of speech mode of the betrayer according to the timeline, i.e., imbalance of speech pattern before and after the decision to betray. This can solve difficult problems, for example, some people in nature are not as polite as the other party of the conversation.
In addition to the logistic regression which is often used for binary classification, support vector machine(SVM) is another classic algorithm that can be used for classification. As they have different advantages, we can design a control group in the experiment to choose the best classifier. Similarly, the semantic features can also be experimented in a controlled manner, so as to select the optimal combination of linguistic features that are most efficient for predicting betrayal.

Read More

Review:

Niculae, Vlad, et al. “Linguistic harbingers of betrayal: A case study on an online strategy game.” arXiv preprint arXiv:1506.04744 (2015).

Summary

The authors are trying to find linguistic cues that can signal revenge. To this end, they use online data for a game called “Diplomacy”. The users are anonymous. The dataset is ideal in that friendships, betrayals and enmities are formed and there is a lot of communication between the users. The authors are interested on Dyad communications, between two people. Subsequently, this textual communication might provide some verbal cues for an upcoming betrayal. As the authors suggest, they do find that certain linguistic cues, such as politeness, leads to betrayals.

Reflections

I have never played the game and I had to read the rules in order to understand it. The research focus is on Dyad communication and betrayal. But are the conversations public, or private? Can the players see you communicating with someone? To get a better understanding I read the rules of the game and the answer is as follows: “In the negotiation phase, players communicate with each other to discuss tactics and strategy, form alliances, and share intelligence or spread disinformation about mutual adversaries. Negotiations may be made public or kept private. Players are not bound to anything they say or promise during this period, and no agreements of any sort are enforceable.”[1]. Subsequently, there is an extra layer of choice besides communication, on whether the negotiations are private or public. These choices might be not captured on Dyad communications.

The authors make a serious point with respect to game theoretic decision making models that attempt to model decision making and interactions. However, I believe that the approach of the authors and the game theoretic approaches are complementary and do not necessarily contradict each other. “Decision theory” is a highly abstract, mathematical, discipline and the models are rigorous in the sense that they follow the scientific method of hard sciences. In addition, just for clarification, the vanilla Prisoners Dilemma is not a “repeated game” as the authors stress. It can’t be formulated as a repeated game however the Nash Equilibrium changes when that happens. Something that should be stressed is that this is not a “repeated game”. The online game was not repeatedly played with the same players over and over again. Had the players played the game repeatedly, eventually they might have changed their behavioral strategies. This in turn might have affected the linguistic cues. In game theory, repeated games, as in games with the same agents and rules that are played over and over again, have different equilibria than static games. Subsequently, I am not sure if the method can be even generalized for this particular game, let alone other games where the rules are different for this reason.

Idiosyncrasy/rules of the game. Eventually, in order to win you need to capture all the territories. Thus the players anticipate that you might eventually become their enemy. This naturally has an impact on the interactions. In the real world, you don’t expect – at least me – to be surrounded by enemies who want to “conquer your territories”. The game induces “betrayal incentives”. If we design the game differently, it is likely that the linguistic predictive features that signal “betrayal” will change. There is a field called “Mechanism Design” dedicated to see how changing the rules of a game yield different results (“equilibria”).

The authors focus on friendships that have at least two consecutive and reciprocated acts of friendship. Should all acts of friendship count the same way? Are some acts of friendship more important than other acts? In other words, should there be a “weight” on an act of friendship?

The authors focus on the predictive factors of betrayal. I wonder, how can we use this in order to inform people on how to maintain friendships. The article makes an implicit assumption that friendships necessarily end due to betrayals. This is natural, because these terms used in the content of “Diplomacy” (the game). In the real world, there could many reasons why friendships can end. It would be interesting to develop a predictive, behavioral, algorithm that predicts the end of friendships because of misunderstandings.

The authors are trying to understand the linguistic aspects of betrayal and as a result, they do not use game specific information. However, if this information is not taken into account, then it is likely that the model will be wrong. By controlling for these effects, we can have a clearer picture of the linguistic aspects of betrayal.

Questions

1. What if the players read this paper before they play the game? Would this change their linguistic cues?
2. Should all acts of friendship count the same way? Are some acts of friendship more important than other acts?
3. What if the authors did control for game specific information as well? Would this alter the results? Based on some models for the same game, it seems that if you select some countries, you will ultimately have to betray your opponent. For instance, “adjacency” is apparently an important factor that determines friendships and enmities. An adjuscency map can be seen in the attached map.[2]
4. What if the users knew each other and played the game again and again, rendering the game repeated? Would this change the linguistic cues, after obtaining information regarding the behavioral patterns from the previous rounds?
5. Can players visually see other players interacting and the length of that interaction? What if they can?

[1] Wikipedia

[2] http://vizual-statistix.tumblr.com/post/64876756583/i-would-guess-that-most-diplomacy-players-have-a

Read More

In very much the same vein as The Language that Gets People to Give: Phrases that Predict Success on Kickstarter by Mitra et al., the authors in this paper look for linguistic cues that foretell betrayal in relationships. Their research focuses on the online game Diplomacy that is set in the pre-World War 1 era. An important aspect of this paper is understanding the game and its intricacies. Each player chooses a country, forms alliances with other players, and tries to win the game by capturing different territories in Europe. Central to the game are these alliances and betrayals, and the conversations that happen when a player becomes disloyal to a friend.

The paper uses draws on prior research work in extracting politeness, sentiment, and linguistic cues for several of its features, and it was instructive to see the uses of some of these social computing tools in their research.

The authors find that there are subtle signs that predict betrayal, namely:

1. An imbalance of positive sentiment before the betrayal, where the betrayer uses more positive sentiment;
2. Less argumentation and discourse from the betrayer;
3. Less planning markers in the betrayer’s language;
4. More polite behavior from the betrayer; and
5. An imbalance in the number of messages exchanged

Intuitively, I can relate to observations #2, #3, and #5. However, positive sentiment and polite behavior would perhaps not indicate betrayal in an offline context. I do wish that these results were explained better and more examples given to indicate why they made sense.

I also felt that the machine learning model to predict betrayal could have been described better. I could not immediately understand their feature extraction mechanism — were linguistic cues used as binary features or count features? Assuming it wasn’t a thin-slicing study and they used count features, did they normalize the counts over the number of times two players spoke? Additionally, they compared the performance of their model against the players (who were never able to predict a betrayal, i.e. their accuracy was 0%). While 0% -> 57% seems like a big jump, the machine learning model could have predicted at random and still obtained a 50% accuracy rate. This begs the question of how accurate the model really is and what features it found important.

Papers in computational social science often need to define (otherwise abstract) social constructs precisely, and quantitatively. Niculae et al. attempt to define friendships, alliances, and betrayals in this paper. While I like and agree with their definitions with respect to the game, it is important to recognize that these definitions are not necessarily generalizable. The paper studies a small subset of relationships online. I would be interested in seeing how this could be replicated for more offline contexts.

Read More