1820 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA...

Detecting Stress Based on Social Interactionsin Social Networks

Huijie Lin, Jia Jia, Jiezhong Qiu, Yongfeng Zhang, Guangyao Shen, Lexing Xie,

Jie Tang, Ling Feng, and Tat-Seng Chua

Abstract—Psychological stress is threatening people’s health. It is non-trivial to detect stress timely for proactive care. With the

popularity of social media, people are used to sharing their daily activities and interacting with friends on social media platforms, making

it feasible to leverage online social network data for stress detection. In this paper, we find that users stress state is closely related to

that of his/her friends in social media, and we employ a large-scale dataset from real-world social platforms to systematically study the

correlation of users’ stress states and social interactions. We first define a set of stress-related textual, visual, and social attributes from

various aspects, and then propose a novel hybrid model - a factor graph model combined with Convolutional Neural Network to

leverage tweet content and social interaction information for stress detection. Experimental results show that the proposed model can

improve the detection performance by 6-9 percent in F1-score. By further analyzing the social interaction data, we also discover several

intriguing phenomena, i.e., the number of social structures of sparse connections (i.e., with no delta connections) of stressed users is

around 14 percent higher than that of non-stressed users, indicating that the social structure of stressed users’ friends tend to be less

connected and less complicated than that of non-stressed users.

Index Terms—Stress detection, factor graph model, micro-blog, social media, healthcare, social interaction

Ç

1 INTRODUCTION

1.1 Motivation

PSYCHOLOGICAL Stress is Becoming a Threat to People’s HealthNowadays.With the rapid pace of life,more andmore peo-

ple are feeling stressed. According to a worldwide surveyreported by Newbusiness in 2010,1 over half of the populationhave experienced an appreciable rise in stress over the lasttwo years. Though stress itself is non-clinical and common inour life, excessive and chronic stress can be rather harmful topeople’s physical and mental health. According to existingresearch works, long-term stress has been found to be relatedto many diseases, e.g., clinical depressions, insomnia etc..Moreover, according to Chinese Center for Disease Control

and Prevention, suicide has become the top cause of deathamong Chinese youth, and excessive stress is considered tobe a major factor of suicide. All these reveal that the rapidincrease of stress has become a great challenge to humanhealth and life quality.

Thus, there is significant importance to detect stress beforeit turns into severe problems. Traditional psychological stressdetection is mainly based on face-to face interviews, self-report questionnaires or wearable sensors. However, tradi-tional methods are actually reactive, which are usually labor-consuming, time-costing and hysteretic. Are there any timelyand proactivemethods for stress detection?

The Rise of Social Media is Changing People’s Life, as Well asResearch in Healthcare and Wellness. With the development ofsocial networks like Twitter and Sina Weibo,2 more andmore people are willing to share their daily events andmoods, and interact with friends through the social net-works. As these social media data timely reflect users’ real-life states and emotions in a timely manner, it offers newopportunities for representing, measuring, modeling, andmining users behavior patterns through the large-scalesocial networks, and such social information can find its the-oretical basis in psychology research. For example, [7]found that stressed users are more likely to be socially lessactive, and more recently, there have been research effortson harnessing social media data for developing mental andphysical healthcare tools. For example, [27] proposed toleverage Twitter data for real-time disease surveillance;while [35] tried to bridge the vocabulary gaps betweenhealth seekers and providers using the community gener-ated health data. There are also some research works [28],[47] using user tweeting contents on social media platforms

1. http://tinyurl.com/htunr9g

� H. Lin, J. Jia, J. Qiu, and G. Shen are with the Department of ComputerScience and Technology, Tsinghua University, Key Laboratory of Perva-sive Computing, Ministry of Education, and Tsinghua National Labora-tory for Information Science and Technology (TNList), Beijing 100084,China. E-mail: {linhuijie, xptree, thusgy2012}@gmail.com, [email protected].

� Y. Zhang is with the University of Massachusetts Amherst, Amherst, MA01003. E-mail: [email protected].

� L. Xie is with the Research School of Computer Science, AustralianNational University, Canberra, ACT 0200, Australia.E-mail: [email protected].

� J. Tang and L. Feng are with the Department of Computer Science andTechnology, Tsinghua University, Beijing 100084, China.E-mail: [email protected], [email protected].

� T.-S. Chua is with the Institute of Systems Science, National University ofSingapore, Singapore. E-mail: [email protected].

Manuscript received 11 Feb. 2016; revised 2 Mar. 2017; accepted 6 Mar. 2017.Date of publication 22 Mar. 2017; date of current version 3 Aug. 2017.Recommended for acceptance by F. Bonchi.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TKDE.2017.2686382

2. http://www.weibo.com, one of the most popular social mediaplatforms in China.

1820 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 29, NO. 9, SEPTEMBER 2017

1041-4347� 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

to detect users’ psychological stress. Existing works [28],[47] demonstrated that leverage social media for healthcare,and in particular stress detection, is feasible.

Limitations Exist in Tweeting Content Based Stress Detection.First, tweets are limited to a maximum of 140 characters onsocial platforms like Twitter and Sina Weibo, and users donot always express their stressful states directly in tweets.Second, users with high psychological stress may exhibitlow activeness on social networks, as reported by a recentstudy in Pew Research Center.3 These phenomena incur theinherent data sparsity and ambiguity problem, which mayhurt the performance of tweeting content based stress detec-tion performance. For illustration, let’s see a Sina Weibotweet example in Fig. 1. The tweet contains only 13 charac-ters, saying that the user wished to go home for the SpringFestival holiday. Although no stress is revealed from thetweet itself, from the follow-up interactive comments madeby the user and her friends, we can find that the user is actu-ally stressed from work. Thus, simply relying on a user’stweeting content for stress detection is insufficient.

Users’ Social Interactions on Social Networks Contain UsefulCues for Stress Detection. Social psychological studies havemade two interesting observations. The first is mood conta-gion [37]: a bad mood can be transferred from one person toanother during social interaction. The second is linguistic ech-oes [34]: people are known to mimic the style and affect ofanother person. These observations motivate us to expandthe scope of tweet-wise investigation by incorporating fol-low-up social interactions like comments and retweetingactivities in user’s stress detection. This may actually help tomitigate the single user’s data sparsity problem. Another rea-son for considering social interactions in stress detection isbased on our empirical findings on a large-scale datasetcrawled from SinaWeibo that the social structures of stressedusers are less connected and thus less complicated than thoseof non-stressed users. This is consistent with the PewResearch Center’s finding that stressed users are less activethan non-stressed ones. The bottom of Fig. 2 illustrates foursocial interaction structure patterns. Each node in a structurepattern represents a user’s interacting friend (who eithercommented or retweeted the tweets). If two nodes are alsofriends on social network, there is an edge linking both; other-wise, there is none. We examined 3,000 users on Sina Weibo.For each user, we collected and merged his/her one weektweets into one and sense stress from it. Meanwhile, we cap-tured the top-3 most active friends the user interacted with.

As shown in Fig. 2, stressed users’ interaction structures areless connected, and thus less complicated than those of non-stressed users.

1.2 Our WorkInspired by psychological theories, we first define a set ofattributes for stress detection from tweet-level and user-level aspects respectively: 1) tweet-level attributes from con-tent of user’s single tweet, and 2) user-level attributes fromuser’s weekly tweets. The tweet-level attributes are mainlycomposed of linguistic, visual, and social attention (i.e.,being liked, retweeted, or commented) attributes extractedfrom a single-tweet’s text, image, and attention list. Theuser-level attributes however are composed of: (a) postingbehavior attributes as summarized from a user’s weekly tweetpostings; and (b) social interaction attributes extracted from auser’s social interactions with friends. In particular, thesocial interaction attributes can further be broken into: (i)social interaction content attributes extracted from the contentof users’ social interactions with friends; and (ii) social inter-action structure attributes extracted from the structures ofusers’ social interactions with friends.

To maximally leverage the user-level information as wellas tweet-level content information, we propose a novelhybrid model of factor graphmodel combined with a convo-lutional neural network (CNN). This is because CNN iscapable of learning unified latent features from multiplemodalities, and factor graph model is good at modeling thecorrelations. The overall steps are as follows: 1) we firstdesign a convolutional neural network (CNN) with crossautoencoders (CAE) to generate user-level content attributesfrom tweet-level attributes; and 2) we define a partially-labeled factor graph (PFG) to combine user-level social inter-action attributes, user-level posting behavior attributes andthe learnt user-level content attributes for stress detection.

We evaluate the proposed model as well as the contribu-tions of different attributes on a real-world dataset from SinaWeibo. Experimental results show that by exploiting theusers’ social interaction attributes, the proposed model canimprove the detection performance (F1-score) by 6-9 percentover that of the state-of-art methods. This indicates that theproposed attributes can serve as good cues in tackling thedata sparsity and ambiguity problem. Moreover, the pro-posed model can also efficiently combine tweet content andsocial interaction to enhance the stress detection performance.

We further conduct in-depth studies on a large-scaledataset from Sina Weibo. Beyond user’s tweeting contents,

Fig. 1. Sample tweets from Sina Weibo. In each tweet, the top part istweet content with text and an image; the bottom part shows the socialinteractions of tweets where there are multiple indicators of stress: men-tions of ‘busy’ and ‘stressed’, ‘working overtime’, ‘failed the exam’,‘money’, and a stressed emoticon.

Fig. 2. The sampling test results of the diversity of users’ social struc-tures from Sina Weibo, by using the top 3 interacted friends of the users.

3. Social Media and the Cost of Caring, 2015, http://www.pewinternet.org/files/2015/01/PI_Social-media-and-stress_0115151.pdf

LIN ETAL.: DETECTING STRESS BASED ON SOCIAL INTERACTIONS IN SOCIAL NETWORKS 1821

we analyze the correlation of users’ stress states and theirsocial interactions on the networks, and address the prob-lem from the standpoints of: (1) social interaction content, byinvestigating the content differences between stressed andnon-stressed users’ social interactions; and (2) social interac-tion structure, by investigating the structure differences interms of structural diversity, social influence, and strong/weak tie. Our investigation unveils some intriguing socialphenomena. For example, we find that the number of socialstructures of sparse connection (i.e., with no delta connec-tions4) of stressed users is around 14 percent higher thanthat of non-stressed users, indicating that the social struc-ture of stressed users’ friends tend to be less connected andcomplicated, compared to that of non-stressed users.

The contributions of this paper are as following.

� We propose a unified hybrid model integrating CNNwith FGM to leverage both tweet content attributesand social interactions to enhance stress detection.

� We build several stressed-twitter-posting datasets bydifferent ground-truth labeling methods from sev-eral popular social media platforms and thoroughlyevaluate our proposed method on multiple aspects.

� We carry out in-depth studies on a real-world large-scale dataset and gain insights on correlations betweensocial interactions and stress, as well as social struc-tures of stressed users.

The rest of this paper is organized as follows. Section 2gives an overview of related works. Section 3 presentsour problem statement. Then in Section 4, we introduce thedefinitions of the proposed attributes. Section 5 presents thehybrid model and training method for stress detection.Experimental results are shown in Section 6. Then in Sec-tion 7, we present several in-depth studies on our datasetfor further insights. Finally, we make some conclusions anddiscuss in Section 8.

2 RELATED WORK

Psychological stress detection is related to the topics of sen-timent analysis and emotion detection.

Research on Tweet-Level EmotionDetection in Social Networks.Computer-aided detection, analysis, and application of emo-tion, especially in social networks, have drawn much atten-tion in recent years [8], [9], [28], [41], [52], [53]. Relationshipsbetween psychological stress and personality traits can be aninteresting issue to consider [11], [16], [43]. For example, [1]providing evidence that daily stress can be reliably recog-nized based on behavioral metrics from users mobile phoneactivity. Many studies on social media based emotion analy-sis are at the tweet level, using text-based linguistic featuresand classic classification approaches. Zhao et al. [53] pro-posed a system calledMoodLens to perform emotion analysison the Chinese micro-blog platform Weibo, classifying theemotion categories into four types, i.e., angry, disgusting, joy-ful, and sad. Fan et al. [9] studied the emotion propagationproblem in social networks, and found that anger has a stron-ger correlation among different users than joy, indicating thatnegative emotions could spread more quickly and broadly inthe network. As stress is mostly considered as a negativeemotion, this conclusion can help us in combining the social

influence of users for stress detection. However, these workmainly leverage the textual contents in social networks. Inreality, data in social networks is usually composed ofsequential and inter-connected items from diverse sourcesandmodalities, making it be actually cross-media data.

Research on User-Level Emotion Detection in Social Networks.While tweet-level emotion detection reflects the instant emo-tion expressed in a single tweet, people’s emotion or psycho-logical stress states are usually more enduring, changing overdifferent time periods. In recent years, extensive researchstarts to focus on user-level emotion detection in social net-works [29], [36], [38], [50]. Our recent work [29] proposed todetect users psychological stress states from social media bylearning user-level presentation via a deep convolution net-work on sequential tweet series in a certain time period.Moti-vated by the principle of homophily, [38] incorporated socialrelationships to improve user-level sentiment analysis inTwitter. Though some user-level emotion detection studieshave been done, the role that social relationships plays in one’spsychological stress states, and how we can incorporate such infor-mation into stress detection have not been examined yet.

Research on Leveraging Social Interactions for Social MediaAnalysis. Social interaction is one of the most important fea-tures of social media platforms. Now many researchers arefocusing on leveraging social interaction information to helpimprove the effectiveness of social media analysis. Fischerand Reuber [12] analyzed the relationships between socialinteractions and users’ thinking and behaviors, and foundout that Twitter-based interaction can trigger effectual cogni-tions. Yang et al. [49] leveraged comments on Flickr to helppredict emotions expressed by images posted on Flickr.How-ever, theseworkmainly focused on the content of social inter-actions, e.g., textual comment content, while ignoring theinherent structural information like how users are connected.

3 PROBLEM FORMULATION

Before presenting our problem statement, let’s first definesome necessary notations.

Let V be a set of users on a social network, and let jV jdenote the total number of users. Each user vi 2 V posts aseries of tweets, with each tweet containing text, image, orvideo content; the series of tweets contribute to users socialinteractions on the social network.

Definition 1 (Stress state). The stress state y of user vi 2 V attime t is represented as a triple ðy; vi; tÞ, or briefly yti. In thestudy, a binary stress state yti 2 f0; 1g is considered, whereyti ¼ 1 indicates that user vi is stressed at time t, and yti ¼ 0indicates that the user is non-stressed at time t, which can beidentified from specific expressions in user tweets or clearlyidentified by user himself, as explained in the experiments. LetY t be the set of stress states of all users at time t.

Definition 2 (Time-varying user-level attribute matrix).Each user in V is associated with a set of attributes A. Let Xt

be a jV j � jAj attribute matrix at time t, in which every row xtixti

corresponds to a user, each column corresponds to an attribute,and an element xt

i;j is the jth attribute value of user vi at time t.

A user-level attribute matrix describes user-specific features, and can be defined in different ways. This study consid-ers user-level content attributes, statistical attributes, andsocial interaction attributes. A detailed discussion of thematrix can be found in Section 4.4. Meaning that three points are connected with each other.


Definition 3 (Time-varying edge set). Users are linked byedges of certain types. Let Et � V � V � C be a set of edgesbetween users at time t. Three types of edges are considered inthe study. For an edge e ¼ ðvi; vj; cÞ 2 Et, c ¼ 0 indicates thatvi follows or is followed by vj at time t, c ¼ 1 indicates thatthere are positive words in comments between user vi and vj attime t, and c ¼ 2 indicates that there are negative words incomments between them at time t.

Definition 4 (Time-varying attribute-augmented net-work). An attribute-augmented network at time t is comprisedof four elements, including 1) a user set V t, 2) an edge set Et, 3)a user-level attribute matrix set Xt, and 4) a stress state set forall users Y t at time t, denoted as Gt ¼ ðV t; Et;Xt; Y tÞ.Given a sequence of labeled time-varying attribute-aug-

mented networks at different times, our goal is to learn amodel that can best fit the relationships among users’ stressstates, user-level attributes, and users’ social linkage, andthen detect users’ unknown stress states with the model.

Problem 1 (Psychological stress detection). Given a seriesof T partially labeled time-varying attribute-augmented net-works fGt ¼ ðV t

L; VtU ; E

t; Y tLÞ j t 2 f1; 2; . . . ; Tgg, V t

L is a setof users with labeled stress states Y t

L at time t, and V tU is a set of

unlabeled users, the objective is to learn a function

f : fG1; G2; . . .GTg ! fY 1U ; Y

2U ; . . .Y

TU g

to predict unlabeled users’ stress states.

4 ATTRIBUTES CATEGORIZATION AND DEFINITION

To address the problem of stress detection, we first definetwo sets of attributes to measure the differences of thestressed and non-stressed users on social media platforms:1) tweet-level attributes from a user’s single tweet; 2) user-level attributes summarized from a user’s weekly tweets.

4.1 Tweet-Level AttributesTweet-level attributes describe the linguistic and visual con-tent, as well as social attention factors (being liked, com-mented, and retweeted) of a single tweet.

For linguistic attributes, we take the most commonlyused linguistic features in sentiment analysis research. Spe-cifically, we first adopt LTP [4]—A Chinese Language Tech-nology Platform—to perform lexical analysis, e.g., tokenizeand lemmatize, and then explore the use of a Chinese LIWCdictionary—LIWC2007 [14], to map the words into posi-tive/negative emotions. LIWC2007 is a dictionary whichcategorizes words based on their linguistic or psychologicalmeanings, so we can classify words into different categories,e.g., positive/negative emotion words, degree adverbs.We have also tested other linguistic resources includingNRC5 and HowNet,6 and found that the performances wererelatively the same, so we adopted the commonly usedLIWC2007 dictionary for experiments. Furthermore, weextract linguistic attributes of emoticons (e.g., and ) andpunctuation marks (‘!’, ‘?’, ‘...’, ‘.’). Weibo defines everyemoticon in square brackets (e.g., they use [haha] for“laugh”), so we can map the keyword in square brackets tofind the emoticons. Twitter adopts Unicode as the represen-tation for all emojis [15], [24], which can be extracteddirectly. The list of linguistic attributes and descriptions areshown in Table 1.

As for the visual attributes, we use API from OpenCV7 toperform picture processing and color-related attributescomputation, e.g., saturation, brightness, warm/cool color,clear/dull color in Table 1. For a special class of attributesnamed five-color theme, we adopt algorithm from paperson affective image classification [32] and color psychologytheories [23], [45]. In this work, we did not adopt the directemotional detection results as visual features because weneed multi-dimensional visual features for deep modellearning, while a direct visual emotional classification resultonly gives a single or very few dimensions as features.However, with the development of emotion-sensitive visualrepresentation techniques, it would be possibility to adoptautomatic visual features in the future. The details of tweet-level attributes are summarized in Table 1.

TABLE 1Summary of Tweet-Level Attributes

Category Short Name # Description

Linguistic Positive & Negative EmotionWords 2 Number of positive and negative emotion wordsPositive & Negative Emoticons 2 Number of popular positive and negative emoticons, e.g., andPunctuation Marks & AssociatedEmotionWords

4 To signify the intensity of emotion four typical punctuation marks(‘!’, ‘?’, ‘...’, ‘.’) are considered.

Degree Adverbs & Associated Emo-tion Words

2 In examples “{I feel a little bit sad}” and “{I feel terribly sad}” , ‘sad’ expressesdifferent negative feelings. We use 1-3 to represent neutral, moderate, andsevere degree of positive emotions, and the minus to represent the negativeones.

Visual Five-color theme 15 A combination of five dominant colors in HSV color space, indicating maincolor distribution of images, has been revealed to be important on human emo-tions by psychology and art theories.

Saturation 2 The mean value of saturation and its contrast.Brightness 2 The mean value of brightness and its contrast.Warm/Cool color 1 Ratio of cool colors with hue ([0-360]) in the HSV space in [30, 110].Clear/Dull color 1 Ratio of colors with brightness ([0-1]) and saturation < 0.6.

Social Social Attention 3 Number of comments, retweets, and likes

The column “#“ indicates the feature vector length for each type of feature.

5. http://www.saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

6. http://www.keenage.com7. http://opencv.org


4.2 User-Level AttributesCompared to tweet-level attributes extracted from a singletweet, user-level attributes are extracted from a list of user’stweets in a specific sampling period. We use one week as thesampling period in this paper. On one hand, psychologicalstress often results from cumulative events or mental states.On the other hand, users may express their chronic stressin a series of tweets rather than one. Besides, the aforemen-tioned social interaction patterns of users in a period oftime also contain useful information for stress detection.Moreover, as aforementioned, the information in tweets islimited and sparse, we need to integrate more complemen-tary information around tweets, e.g., users’ social interac-tions with friends.

Thus, appropriately designed user-level attributes can pro-vide amacro-scope of a user’s stress states, and avoid noise ormissing data. Here, we define user-level attributes from twoaspects to measure the differences between stressed and non-stressed states based on users’ weekly tweet postings: 1) user-level posting behavior attributes [29] from the user’s weeklytweet postings; and 2) user-level social interaction attributesfrom the user’s social interactions beneath his/her weeklytweet postings. The details of user-level attributes are summa-rized in Table 2.

5 MODEL FRAMEWORK

Two challenges exist in psychological stress detection.1) How to extract user-level attributes from user’s tweeting seriesand deal with the problem of absence of modality in the tweets? 2)

How to fully leverage social interaction, including interactioncontent and structure patterns, for stress detection? To tacklethese challenges, we propose a novel hybrid model by com-bining a factor graph model with a convolutional neuralnetwork (CNN), since CNN is capable of learning unifiedlatent features from multiple modalities, and factor graphmodel is good at modeling the correlations. In this section,we will first introduce the architecture of our model, andthen describe the details of each part of the proposed model.

5.1 ArchitectureFig. 3 shows the architecture of our model. There are threetypes of information that we can use as the initial inputs,i.e., tweet-level attributes, user-level posting behavior attrib-utes, and user-level social interaction attributes, whosedetailed computation will be described later. We addressthe solution through the following two key components:

� First, we design a CNN with cross autoencoders(CAE) to generate user-level interaction content attrib-utes from tweet-level attributes. The CNN has beenfound to be effective in learning stationary local attrib-utes for series like images [3], [6] and audios [30], [48].

� Then, we design a partially-labeled factor graph (PFG)to incorporate all three aspects of user-level attributesfor user stress detection. Factor graph model has beenwidely used in social network modeling [10], [39],[44]. It is effective in leveraging social correlations fordifferent prediction tasks.

TABLE 2Summary of User-Level Attributes

Category Short Name # Description

Posting Behavior

Social Engagement 3 The numbers of @-mentions, @-retweets, and @-replies in weekly

tweet postings, indicating one’s social interaction activeness with

friends.

Tweeting time 24 The numbers of tweets posted in hours with a 24-dimensional vector.

Tweeting type 4 Categorize users’ tweets into mainly four types based on general cate-

gories of social media platforms:

(1) Image tweets (tweets containing images);

(2) Original tweets (tweets that are originally authored and posted by

the user);

(3) Information query tweets (tweets that ask questions or ask for help );

(4) Information sharing tweets (tweets that contain outside hyperlinks).

We use a 4-dimensional vector of the numbers of tweets in the above 4

types respectively to quantify the tweeting type attribute.

Tweeting linguistic style 10 Adopt 10 categories from LIWC that are related to daily life, social

events, e.g., personal pronouns, home, work, money, religion, death,

health, ingestion, friends, and family. We extract words from users’

weekly tweet postings, and use a 10-dimensional vector of numbers of

words in the 10 categories

Social Interaction

Content Style

Words 10 A 10-dimensional integer vector, with each value representing the

number of words from social interaction content of users weekly tweet

postings in each word category from LIWC;

Emoticons 2 A 2-dimensional integer vector with each value representing the num-

ber of positive and negative emoticons (e.g., and ) in tweets.

Social Influence

Stressed Neighbor Count 1 The number of the user’s stressed neighbors.

Strong-tie Count 1 The number of stressed neighbors with strong tie.

Weak-tie Count 1 The number of stressed neighbors with weak tie.

Follower Count 1 The number of the user’s followers.

Fans Count 1 The number of the user’s fans.

Social Structure 8 Representing the structure distribution of the user’s interacted friends,

where each element refers to the existence of the corresponding struc-

ture in Fig. 6.

The column “#“ indicates the feature vector length for each type of feature.


Take the user labeled with a red star in Fig. 3 as an exam-ple. We extract attributes from each tweet of the user to formtweet-level attributes as shown in the cylinders. Different col-ors represent different modalities and blank (white color) rep-resents modalities that are not available in the tweet. Thetweet-level attributes in the cylinder are fed to cross autoen-coders (CAEs) [28]. The CAEs are embedded in a CNN [26],[29] that will integrate attributes from CAEs into the aggre-gated user-level content attributes by pooling each attributemap. The user-level content attributes, user-level postingbehavior attributes, and user-level social interaction attributestogether form the user-level attributes. The user-level attrib-utes of a user at time t are denoted by xt

i (i ¼ 1; 2; . . .) in Fig. 3.The route of the other users’ attributes in Fig. 3 are similar,which finally form their user-level attributes. We focus on theattribute flow of the user with red star and omit the detailedroute of other users’ attributes in the figure. The stress state ofeach user at time t is denoted by yti (i ¼ 1; 2; . . .), respectively.The user-level attributes and the stress states are connectedby an attribute factor, while stress states of different users areconnected by social factors. Stress states of the same user atadjacent times are connected by dynamic factors. We definethe graph as a (PFG). By calculating the factors, we can finallyderive all users’ stress states over different weeks.

In the following, we will describe the details of the CNNwith CAE and PFG used in the architecture that tacklesthe tweet series with cropped modalities and leverages thesocial interaction information between users, respectively.

5.2 Learning Aggregated Attributes From TweetSeries

To aggregate user-level attributes, we need to face twomajor challenges: (1) Missing modality, e.g., tweets withonly text but no picture AND (2) How to generate a distrib-uted and modality-invariant representation for each tweets.

To solve above challenges in cross-media tweet data, weuse a cross auto-encoder (CAE) [28] to learn the modality-invariant representation of each single tweet with differentmodalities. Denoting the text, visual, and social attributes ofa tweet by vT , vI , and vS , the CAE is formulated as follows:

u ¼ fðwTvT þ wIvI þ wSvS þ bÞevT ; evI; evSð Þ ¼ fðewuþ ebÞ;�

(1)

where u is the modality-invariant representation. wT , wI ,wS , and b are parameters in the encoder, whereas ewT , ewI ,ewS , and eb are parameters in the decoder. fð�Þ is the activa-tion function. We use a sigmoid activation functionfðzÞ ¼ 1

1þexpð�zÞ in our model. evT , evI , evS are the reconstructed

input modalities.The basic idea of CAE is to force the model to reconstruct

missing modalities in the training stage and to learn crossmodalities correlation from the data (e.g., negative words intext correlate with cool color in pictures). [18] While trainingthe cross auto-encoder, we use training data that contains allthe three modalities. We manually disable the visual modal-ities and/or social interaction8 modality of the trainingdata, and require it to reconstruct all three modalities. Wetrain the CAE with a cropped set of data vT ; vI ; vS thatinputs from one or two modalities are absent, while requir-ing it to reconstruct all the three.

We use the stochastic gradient descent to train the CAE.Denoting all the parameters in the CAE as u, the energyfunction is defined as follows:

J vT ; vT ; vS; uð Þ ¼ 1

2

XM2T;I;S

evM � vMk k2 !

þ �

2

Xm2T;I;S

wMk k2 þ ewMk k2 !

:

(2)

The first term measures the reconstruction accuracy. Thesecond term is the weight decay regularization term thatprevents parameters in the model from diverging arbi-trarily. � is the regularization weight. Using data with dif-ferent modalities as input, the CAE can be trained and learna modality-invariant representation u.

The attributes of tweets, which come from a user’s weeklytweets in timeline, form a time series. To model a user as asubject of series of tweets, we apply CNN [26] which haslarge learning capacity, but has much fewer connections andparameters to learn than similar-size standard networklayers. It focuses on learning stationary local attributes from

Fig. 3. Architecture of our model. The model consists of two parts. The first part is a CNN. The second part is a FGM. The CNN will generate user-level content attributes by convolution with CAE filters as input to the FGM. Take the user labeled with a red star as example. Tweet-level attributes ofthe user are processed through a convolution with CAE to form the user-level content attributes. The user-level attributes are denoted by xti in the leftbox. Every xti contains three aspects: user-level content attributes, user-level posting behavior attributes, and user-level social interaction attributes.Data of other users follows the same route. In the FGM, attribute factors connect user-level attributes to corresponding stress states. Social factorsconnect the stress state of different users. Dynamic factors connect stress state of a user over time. The output of the user’s user-level stress stateat time t is yt1 as highlighted in red, which actually denotes the stress state of the user in weekly period in this paper.

8. Different from the social interaction attributes in this paper, thesocial interaction here is the attribute of a single tweet defined in [28]. Itis simply the mean and variance of interaction numbers of a tweet.


series like images (pixel series), audio, and other time series.We can learn user-level content attributes from a series ofindividual tweets in a time series to describe a user’s stressstate over a week. All attributes of tweets in a time seriesform a one-Dimensional series. We use an 1-DimensionCNN in ourmodel.

CAE units are listed in the attribute maps of the CNN.They connect to a patch of instance. CAE units take patcheswith missing modalities and generate modality-invariantattribute maps. The CAE units are used as filters in the 1-DCNN and convolute over the sequence of tweets to formone feature map. Thus the latent user-level content attrib-utes can be generated from the tweet-level attributes of sin-gle tweets.

Pooling is another important step to summarize attributemaps into fewer attribute instances. Though different usershave different number of tweets in different weeks, theperiod of time over which the tweets are sampled are thesame. We simply pool each attribute map into one pooledattribute. There are two commonly used pooling operations:max-pooling and mean-pooling. When max pooling is used,the pooled attribute unit is assigned with the maximal acti-vation among all units in the attribute map. When mean-pooling is applied, the mean of activations of all units in theattribute map is assigned to the pooled attribute unit. Sincewe pool over the period of time rather than a certain num-ber of tweets, we use mean-over-time (MOT) in this paper,which can be calculated by summing up the activations,since the tweet instances are sampled in the same length oftime intervals.

5.3 Learning Latent Correlations between Tweet’sContent and Social Interactions

As the social correlation between users and time-dependentcorrelation are hard to be modeled using classic classifierssuch as SVM, we use a partially-labeled factor graphmodel (PFG), which was first proposed in [39], to incorpo-rate social interactions and tweets’ content for learning anddetecting user-level stress states.

We define an objective function by maximizing the con-ditional probability of users’ stress states YY given a series ofattribute-augmented networks

GG ¼ fGt ¼ fðV t; Et;Xt; Y tÞgg; t 2 f1; . . . ; Tg;

and V ¼ V 1 ¼ � � � ¼ V T ; jV j ¼ N , i.e., PuðYY jGGÞ. The factorgraph [25] provides a way to factorize the “global” probabil-ity as a product of “local” factor functions, which makes themaximization simple, i.e.,

P ðYY jGGÞ ¼ P ðXX;GjY ÞP ðY ÞP ðXX;GÞ / P ðY jGÞP ðXXjY Þ

/ P ðY jGÞYvi2V

P ðxixijyiÞP ðYY jGGÞ

¼YTt¼1

YNi¼1

fðxtixti; y

tiÞhðyti; ytþ1i Þ

Ye2Et

gðyeÞ;

(3)

The joint probability has three types of factor functions,corresponding to the intuitions we have discussed.

Attribute Factor. We use this factor fðxtixti; ytiÞ to representthe correlation between user vi’s stress state at time t andher/his attributes xxt

i. More specifically, we instantiate the

factor by an exponential-linear function:

fðxxti; y

tiÞ ¼

1

Zaexp aTxxt

i

� �; (4)

where a is a parameter of the proposed model, and Za is anormalization term.

Dynamic Factor. We use this factor fðyti; ytþ1i Þ to representthe time correlation between user vi’s stress state at time tand tþ 1. More specifically, we instantiate the factor by anexponential-linear function:

hðyti; ytþ1i Þ ¼1

Zg

exp gTh0ðyti; ytþ1i Þ� �

; (5)

where g is the model parameters for this type of factor, h0ð�Þis defined as a vector of indicator functions, and Zg is thenormalization term.

Social Factor. We use social factor gðyeÞ (where e ¼ ðvti;vtj; cÞ 2 Et) to represent the correlation between user vi andvj’s stress states according to c at time t:

gðyeÞ ¼ 1

Zbc

exp bcTg0ðyti; ytjÞ

n o; (6)

where bc is the model parameters for this type of factor, g0ð�Þis defined as a vector of indicator functions, and Zbc is thenormalization term.

Finally, by combining Eq. (4), (5), and (6) into Eq. (3), theobjective function as the log-likelihood of the proposedmodel is,

O ¼XTt¼1

XNi¼1

aTxxti þXTt¼1

XNi¼1

gTh0ðyti; ytþ1i Þ

þXTt¼1

Xe2Et

bTc g0ðyti; ytjÞ � logZ;

(7)

where Z ¼ Za

Qc2C ZbcZg is the global normalization term.

Algorithm 1. Learning and Inference by Factor Graph

Input: a series of time-varying attribute augmented network GGwith stress states on some of the user nodes, learningrate h;

Output: parameter value u ¼ fa; fbcg; gg and full stress statevector YY ;

Randomly initialize YY ;Initialize model parameters u;repeatCompute gradientra;rbc;rg ;Update a aþ h�ra;Update bc bc þ h�rbc;Update g g þ h�rg;

until convergence;

Learning. Learning the predictive model is to estimate aparameters configuration u ¼ ða; fbcg; gÞ from the partially-labeled dataset and to maximize the log-likelihood objectivefunction Eq. (7), i.e., u� ¼ argmaxuOðuÞ.

For optimization, we adopt a gradient decent method.Specifically, we derive the gradients with respect to eachparameter in our objective function of Eq. (7)


@O@a¼ E

XTt¼1

XNi¼1

fðxxti; y

tiÞ

" #� EPaðYY jGGÞ

XTt¼1

XNi¼1

fðxxti; y

tiÞ

" #@O@b¼ E

XTt¼1

Xe2Et

gðyeÞ" #

� EPbðYY jGGÞXTt¼1

Xe2Et

gðyeÞ" #

@O@g¼ E

XTt¼1

XNi¼1

hðyti; ytþ1i Þ" #

� EPg ðYY jGGÞXTt¼1

XNi¼1

hðyti; ytþ1i Þ" #

;

(8)

where in the first equation, E½PTt¼1PN

i¼1 fðxxti; ytiÞ� is theexpectation of the summation of the attribute factor func-tions given the data distribution over YY and GG in the train-ing set, and EPaðYY jGGÞ½

PTt¼1PN

i¼1 fðxxti; y

tiÞ� is the expectation

of the summation of the attribute factor functions given bythe estimated model. The other expectation terms have simi-lar meanings in the other equation.

As the network structure in the real world may containcycles, it is intractable to estimate the marginal probability inthe second terms of (8). In this work, we adopt Loopy BeliefPropagation (LBP) [33] to calculate the marginal probabilityof P ðYY Þ and compute the expectation terms. The learningprocess can then be described as an iterative algorithm. Eachiteration contains two steps. First, we call LBP to calculatemarginal distributions of unknown variables PaðYY jGGÞ. Sec-ond, we update a, b, g with the learning rate h by Eq. (9) Thelearning algorithm terminates when it reaches convergence,

unew ¼ uold þ h@O@u

: (9)

Detection. With the estimated parameter u, we can nowassign the value of unknown labels YY by looking for a labelconfiguration that will maximize the objective function, i.e.,

Y � ¼ argmax OðYY jGG; uÞ: (10)

In this paper, we use a max-sum algorithm [31] to solve thisproblem.

6 EXPERIMENTS

In this section, we will present the effectiveness and effi-ciency of our hybrid model on user-level stress detection.

6.1 Dataset CollectionTo conduct observations and evaluate our succecive model,we first collect a set of datasets using different labelingmethods, which are listed as following:

Dataset DB1. It is a challenge to construct a dataset withreliable ground truth labels from large-scale noisy socialmedia data. The data crawled from social platforms is usu-ally massive, thus manual labeling methods are not feasible

due to the uncontrollable cost and quality. To solve this prob-lem, we employed a sentence pattern labeling method toautomatically extract labeled data from the crawled large-scale social media data. We first crawled 350 million tweetsdata via Sina Weibo’s REST APIs9 from Oct. 2009 to Oct.2012. Sina weibo, as the biggest microblog website in China,provides users an open online platform for information shar-ing, communication and obtaining. Similar to Twitter andFacebook, users on Sina Weibo can post contents with multi-ple modalities, including text, image, social action (retweet,comment, favorite), video and etc. Despite these user gener-ated contents, user relationship, which takes the form of fol-lowing on Sina Weibo, also contains abundant informationfor data analysis. Utilizing above information and featuresextracted frommultiplemodalities, we are able to investigateusers emotions, stresses and opinions.

We then tried to identify the weekly stressed state ofusers. Facing the vast scale of social images, manually label-ing is powerless. Instead, we use tags and comments forautomatic image labeling, which is a common method inprevious work [20], [21], [46]. This is done by searching fortweets containing patterns like “I feel stressed this week” and“I feel stressed so much this week”, which are used to indicatethat the users are stressed. The weeks containing such sen-tence patterns are labeled as “stressed” weeks. Similarly,we identify “non-stressed” weeks of users by searching fortweets with patterns like “I feel relaxed” and “I feel non-stressed”. These sentence patterns have been shown to havehigh precision against user-assigned psychological statelabels validated by online surveys in weibo [29].

In this way, we collected over 19,000 weeks of tweets thatare labeled as stressed, and over 17,000 weeks of non-stressed users’ tweets. There are 492,676 tweets from 23,304users in total. We use this dataset for experiments, analysisand further in-depth studies, which is represented by DB1in this paper. Details of the dataset are shown in Table 3.

Dataset DB2. We verified the reliability of the aboveground truth labeling method through dataset DB2 inTable 4. It is a small dataset collected from the users whohave shared the score of a psychological stress scale PSTR10

designed by psychologists via Weibo. Guided by the rulesof the PSTR scale, a user is taken as stressed when the scoreis larger than 80, otherwise non-stressed. We thus crawledthe scores posted by users, and used the scores as groundtruth label for the set of tweets in þ�3-day window.

TABLE 3Overview of the Weibo-Stress Dataset

non-stressed stressed total

#tweets 253,638 239,038 492,676#users 12,230 11,074 23,304#weeks 17,861 19,136 36,997#tweets/week* 14.2 12.5 13.3#weeks/user* 1.46 1.73 1.59#interacting users/week* 5.79 6.99 6.35

* means average number.

TABLE 4Details of Other Datasets

Platform Stress label Numberof tweets

Numberof users

Numberof weeks

Tweetsper week

DB2:Sina Weibo(2010.2-2011.9)

stressed 1,459 98 98 14.9non-stressed 1,845 112 112 16.5summary 3,304 210 210 15.7

DB3:Tencent Weibo(2011.11-2013.3)

stressed 138,570 7,845 8,974 15.4non-stressed 172,585 8,239 9,976 17.3summary 311,155 16,084 18,950 16.4

DB4:Twitter(2009.6-2009.12)

stressed 54,748 4,905 6,081 9.0non-stressed 75,357 4,018 6,545 11.5summary 130,105 8,923 12,626 10.3

9. http://open.weibo.com10. http://types.yuzeli.com/survey/pstr50


Dataset DB3 and DB4. To further test our method, we col-lected two more datasets from Tencent Weibo (DB3) andTwitter (DB4). They are again labeled using the sentencepattern labeling method as described above for DB1. In par-ticular, as social platforms of different languages, Weiboand Twitter have many differences. [51]. For example, theirtop topics differs very much. Thus, experiments on Twittercan validate the universality of our method. The details ofthe two datasets are presented in Table 4.

6.2 Experimental SetupIn the following experiments, we first train and test ourmodel on the large-scale Sina Weibo dataset DB1. We thentest our model on the other 3 datasets to show effectivenessof the proposed model on different data sources or differentground truth labeling methods. For all of our analysis, weuse 5-fold cross validation, with over 10 randomized experi-mental runs.

Comparison Methods.We compare the following classifica-tion methods for user-level psychological stress detectionwith our FGM+CNNmodel (denoted as FGM here).

� Logistic Regression (LRC) [19]: it trains a logisticregression classification model and then predictsusers’ labels in the test set.

� Support Vector Machine (SVM) [5]: it is a popular andbinary classifier that is proved to be effective on ahuge category of classification problems. In ourproblem we use SVMwith RBF kernel.

� Random Forest (RF) [42]: it is an ensemble learningmethod for decision trees by building a set of deci-sion trees with random subsets of attributes and bag-ging them for classification results.

� Gradient Boosted Decision Tree (GBDT) [13]: it trains agradient boosted decision tree model with featuresassociated with each user.

� Deep Neural Network (DNN) [29] for user-level stressdetection: it is proposed to deal with the problem ofuser-level stress detection problem with a convolu-tional neural network (CNN) with cross autoen-coders. This is the real baseline method that we cancompare our proposed model with.

We employ scikit-learn11 for the above methods.Evaluation Measures. For a fully investigation of the pro-

posed methods, we consider the following aspects:

� Effectiveness. We evaluate the detection performanceof our model and comparison methods in terms ofAccuracy (Acc.), Recall (Rec.), Precision (Prec.) andF1-Measure (F1) [2].

� Efficiency. We evaluate efficiency of the methodsby comparing the CPU time of training each model.All experiments are performed on an x64 machinewith 2.9 GHz intel Core i7 CPU and 8 GB RAM.

6.3 Experimental Results on DB1Comparison of Detection Performance. To evaluate the effective-ness of ourmodel, we first conduct a test using different mod-els based on the Weibo-Stress dataset. In this experiment, weused all the three attributes described in previous section:user-level social interaction attributes, user-level posting

behavior attributes and user-level content attributes gener-ated from the tweet-level attributes by CNN+CAE. Table 5shows the experimental results. We see that FGM gains supe-rior results against the comparative methods, which verifiesthat our proposed model can effectively leverage the socialinteraction and social structure attributes for stress detection.Compared with the results in [29], which also aims at user-level stress detection based on social media data sources, ourproposed model improves the detection performance by upto 9 percent on F1-score. These results demonstrate the feasi-bility of stress detection via the brand new information sourceof social interactions, and that our proposedmodel can signif-icantly enhance the performance by leveraging the socialinteraction information.We further perform t-tests and all thep-values are 0:01, indicating that the improvements of ourproposed models over the comparison methods are statisti-cally significant.

Comparison of Model Efficiency. To evaluate the efficiencyof the aforementioned methods, we compare the CPU timeof training each model. The comparison results are alsoshown in Table 5. Overall, all methods have good efficiencyperformance, and the running time of different methodsranges from seconds to minutes. FGM results in a slightlylower but better performance compared to other methods.

Factor Contribution Analysis. The definition of factors isimportant to the performance of the Factor Graph Model. Wehave three types of factors in our model, i.e., attribute factor,social factor, and dynamic factor. To analyze the impact ofdifferent factors in our model, we compare the detection per-formancewith different combinations of factors in this experi-ment, as shown in Fig. 4a. Specifically, we first use all thethree factors, denoted as FGM, then we remove the followingfactors respectively: social factor, dynamic factor and both ofthem, denoted as FGM-S, FGM-D and FGM-S-D. We see thatthe worst performance is achieved if we incorporate only theattribute factor. However, integrating attribute factor withsocial or dynamic factor both improve the performance,revealing that both of the two factors are effective for stressdetection. Specifically, incorporating social factor significantlyimproves the detection performance to around 91 percent onaccuracy, indicating that the social factor is extremely effec-tive. The best detection performance is observed when usingall three types of factors.

Training Data Scale Analysis. To evaluate the data scalabil-ity of the proposed model, we try to train the model withdifferent scale of training data, and compare the final detec-tion performance in F1-score. In this test, we use all thethree attributes as input. Fig. 4b shows the trend of detec-tion performance with different proportions of trainingdata. It is clear that when using only 1 percent of all trainingdata, our model fails to achieve meaningful detection per-formance. When adopting approximately 30 percent of alltraining data, our model can obtain an equally competitiveperformance of around 93 percent compared with thatwhen using 50 percent of training data. Moreover, the per-formance keeps increasing given more training data. Theseresults verify the scalability of our model on large-scalereal-world social media datasets.

Convergence Analysis. We further investigate the conver-gence of the learning algorithm for FGM, and Fig. 4cpresents the F1-score with increasing number of iterations.We see that the algorithm converges within around 2,000iterations, which is rapid enough for us to conduct efficientmodel training on large scale datasets in practice.11. http://scikit-learn.org


Impact of Size of Network. Size of network is a critical issuein setting up DNNmodel. Shallow networks result in trivialmodel that cannot catch any underlying correlation in data,whereas too deep networks lead to over-complex modelwhich is difficult to tune and may suffer from problems likeover-fitting. To choose an appropriate DNN model for clas-sification, we test DNN with different number of layers.Fig. 4d summarizes the experiment results. It is clear that 2-layer is not sufficient for the model to achieve a satisfactoryresult. 3-layer model improve significantly while 4-layermodel reaches the peak. 5-layer model does not get betterresult. This is mainly because at 5-layer the network may betoo large that it cannot be tuned well with the available dataand within a feasible training time.

Attribute Contribution Analysis. As described in Section 4,we have defined several set of tweet-level and user-levelattributes from a single tweet’s content as well as users’posting behaviors and social interactions in a weeklyperiod. To evaluate the contribution of different attributesand compare the effectiveness of our model of leveragingdifferent attributes, we compared the proposed model withother existing models by using different combinations ofattributes as input. As described in Section 4, the proposedattributes are categorized into four groups: tweet-levelattributes, user-level posting behavior attributes, user-levelsocial interaction content attributes and user-level socialinteraction structure attributes, denoted as T, UPB, UIC,and UIS respectively. We compare the detection perfor-mance of the proposed CNN+FGM model with SVM andCNN with traditional autoencoder, with all the possiblecombinations of these four set of attributes. For the SVMwith the tweet-level attributes, we simply take the averageof the feature vectors from a user’s weekly tweets.

The results of this experiment are shown in Fig. 5. We seethat all the models achieve the best detection performancewhen utilizing all the three set of attributes. When usingonly the tweet-level features, the detection performance ofthe proposed model and the DNN model drops to around86 and 82 percent respectively in F1-score, which is accept-able. While for SVM, the detection performance drops to

around only 70 percent, which is poor for a binary classifica-tion. This result demonstrates the effectiveness of the fea-ture aggregation of CNN, which is much better than simplysummarizing the feature vectors manually.

Fig. 5 also shows the effectiveness of different attributes.We can see that by using only user-level attributes, the detec-tion performance of all the models drops drastically com-pared to that using only tweet-level attributes, which showsthe importance of the tweet-level attributes. By combing dif-ferent types of user-level attributes, the detection performanceimproves by around 3-8 percent in F1-score, showing that theuser-level attributes are supplementary to each other. Mean-while, by combining the user-level attributes with tweet-levelattributes, the detection performance improves up to 10-20percent in F1-score. This result indicates that the user-levelattributes are great supplements to tweet-level attributes.

When using only two set of attributes, the detection perfor-mance drops to around 91 percent in F1-score. In case of usingsole attributes, we see that using solely user-level social inter-action attributes gets the best detection performance ofaround 90 percent in F1-score, as compared to the other attrib-utes. This reveals that the proposed user-level social interac-tion attributes are quite effective for stress detection.

Impact of Different Modalities in Content Attributes. Tweetscontent come with multiple modalities. To evaluate the con-tribution of each data modality, we conduct experimentswith different combination of attributes. Since text is thenecessary part of a tweet, we test using solely text attributes,and the two combinations of text and visual attributes, andtext and social attributes, as well as using all attributes. Theresults are shown in Table 6. It is interesting to note thatusing only text attribute could achieve rather high perfor-mance. Simply combining visual or social attributes withtext attributes may even reduce the performance, especiallythe social attributes. This trend is even more obvious whenboth types of attributes (content and posting behavior) areused. Nevertheless, using all attributes together outper-forms that using only the text attributes; and the highestperformance is observed when using all attribute and work-ing with both types of attributes.

6.4 Results on Other DatasetsWe further evaluate our model on other datasets, DB2-DB4,as shown in Table 4, to show that our model is universallyapplicable. For these experiments, we use all the proposedattributes with MOT pooling, and a 4-layer DNNmodel.

DB2 from Sina Weibo with PSTR Label. We use a maturedmodel trained with large scale Sina Weibo dataset, and thentest it against another set of subject independently sampledfrom Sina Weibo. For the test set, we collect weekly tweetsfrom the users that have shared the score of a psychological

Fig. 4. Experiment results analysis from various perspectives. (a) Attribute contribution analysis. (b) Factor contribution analysis. (c) Results ofdetection performance with different training data scales. (d) Convergence analysis of FGM.

TABLE 5Comparison of Efficiency and Effectiveness Using

Different Models (%)

Method Acc. Rec. Prec. F1 CPU time

LRC 76.18 87.94 78.58 83.00 39.43 sSVM 72.58 87.39 75.16 80.82 10 minRF 77.73 89.63 79.35 84.18 67.71 sGBDT 79.75 82.99 85.90 84.43 262.86 sFGM 91.55 96.56 90.44 93.40 20 min


stress scale with 50 items via Sina Weibo. Detection resultshows that the test accuracy is 84.26 percent and F1-score is0.8785, which demonstrates that the overall model is consis-tent and the sentence pattern based ground truth labelingmethod is reliable.

DB3 from Tencent Weibo. We test on data collected fromanother major Chinese social media platform. For this test,we use the attribute extractor trained with large scale SinaWeibo dataset and only finetune the network with Twitterdataset in 5-fold. The accuracy is 86.18 percent and F1-scoreis 0.8832 which demonstrate the capability of the model.

DB4 from Twitter. We also test against the Twitter dataset.We still use the attribute extractor trained with large scaleSinaWeibo dataset and only finetune the network with Twit-ter dataset in 5-fold. The accuracy is 77.43 percent and F1-score is 0.8224. One reason for this modest result is that usersin Twitter dataset and Sina Weibo dataset come from differ-ent language and culture background, so that the languagepatterns and sentimental signals from these two differentlanguage environments can be different, thus the attributeextractor trained with large scale SinaWeibo dataset may notbe fully functional for Twitter datasets. Nevertheless, we stillachieved acceptable performance in Twitter dataset, whichimplies that the basic stress patterns between social relationscan be transferred in between different language environ-ments. Another factor could be that the scale of this dataset israther small. Subjects in the Twitter dataset are on the orderof 10 percent than that in large-scale Sina Weibo dataset. Welook into the collected data and find that, by coincidence, alltweets in this dataset have no social activity. We conjecturethis is also one of the causes of the unsatisfactory result.

7 STUDIES OF SOCIAL INTERACTION

We have presented the experimental results on stress detec-tion in the previous section, while in the setting of social net-works, it would be helpful to further analyze how a user’sstress status is developed and how they affect each other.To do so, we try to conduct several studies on DB1 to offerinsights on how social interactions contribute to user stressand the task of stress detection from the following aspects:

(1) Content. How are users’ social interaction contents (e.g.,language used) related to users’ stress states?

(2) Structure. Compared to non-stressed users, do stressedusers show different structural diversity patterns when theybehave in social networks?Do differences of social influence andstrong/weak ties exist between stressed and non-stressed users?

7.1 ContentContent of social interaction refers to the content of tweets’comments and retweets, including text, emoticons, andpunctuation marks. Based on a widely used psychologicaldictionary LIWC2007 [40], we extract emotional words fromthe interaction content of tweets, and categorize the extractedwords into corresponding groups defined in LIWC2007.We compare the frequencies of different word categoriesbetween stressed and non-stressed users.

Fig. 6 shows the comparison results of the most widelyused word categories in our data set, we observe that there isan obvious difference in interaction contents between stressedand non-stressed users. That is, interaction contents ofstressed users’ tweets contains much more words from cate-gories like death, sadness, anxiety, anger, and negative emo-tion, while non-stressed users’ tweets contain more wordsfromcategories like friends, family, affection, leisure, and pos-itive emotion.

7.2 StructureTo examine structure properties (i.e., social influence andstrong/weak tie) of (non)stressed users, we use risk ratio(RR) to measure the correlation between users’ stress statesand different structural attributes. Risk ratio is an effectivemeasurement widely used in the statistical analysis and rel-evant fields. The risk ratio of a stressed state, associatedwith a structural attribute a, is calculated as follows:

RRðaÞ ¼ P ðstressed user has attribute aÞP ðstressed user does not have attribute aÞ : (11)

A larger risk ratio implies that users with attribute a are morelikely to be stressed. In this section, we investigate representa-tive sociology theories, and quantitatively analyze the correla-tions between users’ stress states and fundamental socialconcepts, so as to examine how andwhy a user’s stress state isdeveloped and affected by other users.

7.2.1 Structural Diversity

We are interested in whether stressed and non-stressedusers have any structural difference in respective friends’connection. In sociology, social structure refers to a society’sframework, consisting of various relationships among peo-ple, as well as groups that direct and set limits on humanbehaviors. In social networks, direct connections (followingor followed) of users that interact with each other via com-ments and retweets also form a kind of social structure. Forthis in-depth study, we select top four users with the mostfrequent interactions from users’ weekly tweet postings,where four is adopted because this is the minimum numberof nodes required to produce structural combinations(10 combinations), so as to calculate the probability of eachcombination, and incorporating more nodes would makethe calculation combinatorial expensive. We measure theconnection of the interacting users by the following link, thatis, if A is following or followed by B, then A and B are con-nected, and cliques made up of different nodes are treated

TABLE 6Comparison of Results Using Different Modalities

Text Text + visual Text + Social All

Accuracy 0.8713 0.8761 0.8628 0.9155F1-score 0.8794 0.8865 0.8711 0.9340

Fig. 5. Experiment results analysis of different attribute combinations ondifferent models, with T, UPB, UIC, and UIS representing tweet-levelattributes, user-level posting behavior attributes, user-level social Inter-action content attributes and user-level social Interaction Structureattributes respectively. For example, ‘UIC+UIS’ here means a combina-tion of user-level social Interaction content attributes and user-levelsocial interaction structure attributes.


the same. We compare the proportion of different socialstructures of interacting users to measure the structuraldiversity. The results in Fig. 7 clearly show us that structuraldifferences do exist between stressed and non-stressed users.The number of social structures of sparse connection (i.e.,with no delta connections) of stressed users is around 14 per-cent higher than that of non-stressed users, indicating thatthe social structure of stressed users’ friends tends to be lessconnected and less complicated, compared to that of non-stressed users. This phenomenon has also been reported bythe current psychological research result that stressed usersare more likely to be socially less active [7].

7.2.2 Social Influence

Social influence is an important factor that governs thedynamics of social networks. The principle of social influ-ence [22] suggests that users tend to change their behav-iors to match their friends’ behaviors. In this study, wetry to examine whether users’ stress states will be influ-enced by their neighbors’ states by looking at the proba-bility of a user’s stress state when he/she has differenttypes of relationships with other stressed users. As forthe stress state labeling, all users including friends arelabeled using the sentence pattern method described inprevious section.

Fig. 8a shows the probability that a user being stressed,conditioned on the number of stressed neighbors that the userhas in the social network. We can see that being stressed is amutually correlated behavior. In particular, the chance that anone-stressed user becoming stressed increases to three timeshigher for those with stressed neighbors than for those with-out. Another trend observed from Fig. 8a is that the likelihood

of a user becoming stressed increases with the number ofstressed neighbors.

7.2.3 Strong/Weak Tie

Strong/Weak Tie [17] is one of the most basic principles insocial network theories. We classify the constructed socialrelationships into strong or weak ties by the number oftimes that two users interact with each other via comment,@-mention, retweet, or like in a week. In our work, we trieddifferent values for the threshold and finally chose three bycross-validation. If two users interact with each other morethan three times, we call the relationship a strong tie, andotherwise a weak tie. This definition of user ties is adoptedas the standard treatment in the research of social networkanalysis [17], so as to capture the most recent user relation-ships in a shifting environment. Fig. 8b illustrates theresults. We can see that strong ties indeed have strong influ-ence on users’ stress states, and the influence of weak ties isrelatively weak. For example, when a user has three stressedstrong-tie connections, the probability that the user willbecome stressed increases to 13 percent, more than twice ashigh as for a user with three stressed weak-tie connections.

Summary. Based on the experimental results and analyseswe know that: 1) users’ stress states are not only revealed intheir own tweets, but also affected by the contents of theirsocial interactions, including commenting on and re-tweet-ing others’ tweets; and 2) users’ stress states are revealed bythe structure of their social interactions, including structuraldiversity, social influence, and strong/weak ties. Theseinsights quantitatively prove the necessity and effectivenessof combining social interactions for stress detection.

8 CONCLUSION

In this paper, we presented a framework for detectingusers’ psychological stress states from users’ weekly socialmedia data, leveraging tweets’ content as well as users’social interactions. Employing real-world social media dataas the basis, we studied the correlation between user’ psy-chological stress states and their social interaction behav-iors. To fully leverage both content and social interactioninformation of users’ tweets, we proposed a hybrid modelwhich combines the factor graph model (FGM) with a con-volutional neural network (CNN).

In this work, we also discovered several intriguing phe-nomena of stress. We found that the number of social struc-tures of sparse connection (i.e., with no delta connections)of stressed users is around 14 percent higher than that ofnon-stressed users, indicating that the social structure ofstressed users’ friends tend to be less connected and less

Fig. 7. Distribution of stress states (stressed and non-stressed) over dif-ferent social structures. The dot represents a friend of the user, and theline represents the connection of friends.

Fig. 6. Distribution of stress states (stressed and non-stressed) over dif-ferent word categories from tweets’ comments and retweets. Here, weshow 10 most widely used word categories in our data set.

Fig. 8. Social influence and Social tie analysis. (a) Variation trend ofprobability of a user being stressed when she/he has different numberof stressed neighbors. (b) Variation trend of probability of a user beingstressed when she/he has different number of stressed neighbors withstrong/weak ties.


complicated than that of non-stressed users. These phenom-ena could be useful references for future related studies.

ACKNOWLEDGMENTS

Thiswork is supported byNational Key Research andDevel-opment Plan (2016YFB1001200), the Innovation MethodFund of China (2016IM010200). This work is also supportedby the National Natural, and Science Foundation of China(61370023, 61521002, 61373022). Besides, this work is sup-ported in part by the Australian Research Council via theDiscovery Project program DP140102185, and also sup-ported by a research fund from MSRA and the RoyalSociety-Newton Advanced Fellowship Award. We wouldalso like to thank “Tsinghua University-Tencent Joint Labo-ratory” for its support. This research is also part of the NExTresearch, supported by the National Research Foundation,Prime Minister’s Office, Singapore under its IRC@SGFunding Initiative.

REFERENCES

[1] A. Bogomolov, B. Lepri, M. Ferron, F. Pianesi, and A. Pentland,“Daily stress recognition from mobile phone data, weather condi-tions and individual traits,” in Proc. ACM Int. Conf. Multimedia,2014, pp. 477–486.

[2] C. Buckley and E. M. Voorhees, “Retrieval evaluation with incom-plete information,” in Proc. 27th Annu. Int. ACM SIGIR Conf. Res.Development Inf. Retrieval, 2004, pp. 25–32.

[3] X. Chang, Y. Yang, A. G. Hauptmann, E. P. Xing, and Y.-L. Yu,“Semantic concept discovery for large-scale zero-shot eventdetection,” in Proc. Int. Joint Conf. Artif. Intell., 2015, pp. 2234–2240.

[4] W. Che, Z. Li, and T. Liu, “Ltp: A chinese language technologyplatform,” in Proc. Int. Conf. Comput. Linguistics, 2010, pp. 13–16.

[5] C. C. Chang and C.-J. Lin, “Libsvm: A library for support vectormachines,” ACMTrans. Intell. Syst. Technol., vol. 2, no. 3, pp. 389–396,2001.

[6] D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, andJ. Schmidhuber, “Flexible, high performance convolutional neuralnetworks for image classification,” in Proc. Int. Joint Conf. Artif.Intell., 2011, pp. 1237–1242.

[7] S. Cohen andA.W. Thomas, “Stress, social support, and the bufferinghypothesis,” Psychological Bulletin, vol. 98, no. 2, pp. 310–357, 1985.

[8] G. Coppersmith, C. Harman, and M. Dredze, “Measuring posttraumatic stress disorder in twitter,” in Proc. Int. Conf. WeblogsSoc. Media, 2014, pp. 579–582.

[9] R. Fan, J. Zhao, Y. Chen, and K. Xu, “Anger is more influentialthan joy: Sentiment correlation in weibo,” PLoS One, vol. 9, 2014,Art. no. e110184.

[10] Z. Fang, et al., “Modelingpaying behavior in game social networks,”inProc. 23rd Conf. Inform. Knowl.Manag., 2014, pp. 411–420.

[11] G. Farnadi, et al., “Computational personality recognition in socialmedia,”UserModel.User-Adapted Interaction, vol. 26, pp. 109–142, 2016.

[12] E. Fischer and A. R. Reuber, “Social interaction via new socialmedia: (How) can interactions on twitter affect effectual thinkingand behavior?” J. Bus. Venturing, vol. 26, no. 1, pp. 1–18, 2011.

[13] J. H. Friedman, “Greedy function approximation: A gradient boost-ingmachine,”Ann. Statist., vol. 29, no. 5, pp. 1189–1232, 1999.

[14] R. Gao, B. Hao, H. Li, Y. Gao, and T. Zhu, “Developing simplifiedchinese psychological linguistic analysis dictionary for micro-blog,” in Proc. Int. Conf. Brain Health Informat., pp. 359–368, 2013.

[15] J. Gettinger and S. T. Koeszegi,More Than Words: The Effect of Emo-ticons in Electronic Negotiations. Berlin, Germany: Springer, 2015.

[16] J. Golbeck, C. Robles, M. Edmondson, and K. Turner, “Predictingpersonality from Twitter,” in Proc. IEEE 3rd Int. Conf. Privacy, Secu-rity, Risk Trust, IEEE 3rd Int. Conf. Soc. Comput., 2011, pp. 149–156.

[17] M. S. Granovetter, “The strength of weak ties,” Amer. J. Sociology,vol. 78, pp. 1360–1380, 1973.

[18] Q. Guo, J. Jia, G. Shen, L. Zhang, L. Cai, and Z. Yi, “Learningrobust uniform features for cross-media social data by using crossautoencoders,” Knowl. Based Syst., vol. 102, pp. 64–75, 2016.

[19] D. W. Hosmer, S. Lemeshow, and R. X. Sturdivant, Applied LogisticRegression. Hoboken, NJ, USA: Wiley, 2013.

[20] S. J. Hwang, “Discriminative object categorization with externalsemantic knowledge,” 2013.

[21] S. D. Kamvar, “We feel fine and searching the emotional web,” inProc. 4th ACM Int. Conf. Web Search DataMining, 2011, pp. 117–126.

[22] H. C. Kelman, “Compliance, identification, and internalization:Three processes of attitude change,” General Information, vol. 1,no. 1, pp. 51–60, 1958.

[23] S. Kobayashi, “The aim and method of the color image scale,”Color Res. Appl., vol. 6, no. 2, pp. 93–107, 1981.

[24] N. P. Kralj, J. Smailovi, B. Sluban, and I. Mozeti, “Sentiment ofemojis,” Plos One, vol. 10, no. 12, 2015, Art. no. e0144296.

[25] F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphsand the sum-product algorithm,” IEEE Trans. Inform. Theory,vol. 47, no. 2, pp. 498–519, Feb. 2001.

[26] Y. LeCun and Y. Bengio, “Convolutional networks for images,speech, and time series,” The Handbook of Brain Theory and NeuralNetworks. Cambridge, MA, USA: MIT Press, 1995.

[27] K. Lee, A. Agrawal, and A. Choudhary, “Real-time disease sur-veillance using twitter data: Demonstration on FLU and cancer,”in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining,2013, pp. 1474–1477.

[28] H. Lin, J. Jia, Q. Guo, Y. Xue, J. Huang, L. Cai, and L. Feng,“Psychological stress detection from cross-media microblog datausing deep sparse neural network,” in Proc. IEEE Int. Conf. Multi-media Expo, 2014, pp. 1–6.

[29] H. Lin, et al., “User-level psychological stress detection fromsocial media using deep neural network,” in Proc. ACM Int. Conf.Multimedia, 2014, pp. 507–516.

[30] L. Liu and L. Shao, “Learning discriminative representations fromRGB-d video data,” in Proc. Int. Joint Conf. Artif. Intell., pp. 1493–1500, 2013.

[31] H.-A. Loeliger, “An introduction to factor graphs,” IEEE SignalProcess. Mag., vol. 21, no. 1, pp. 28–41, Jan. 2004.

[32] J. Machajdik and A. Hanbury, “Affective image classificationusing features inspired by psychology and art theory,” in Proc.Int. Conf. Multimedia, 2010, pp. 83–92.

[33] K. P. Murphy, Y. Weiss, and M. I. Jordan, “Loopy belief propaga-tion for approximate inference: An empirical study,” in Proc. 15thConf. Uncertainty Artif. Intell., 1999, pp. 467–475.

[34] C. D. N. Mizil, L. Lee, B. Pang, and J. Kleinberg, “Echoes of power:Language effects and power differences in social interaction,”eprint arXiv:1112.3670, 2011.

[35] L. Nie, Y.-L. Zhao, M. Akbari, J. Shen, and T.-S. Chua, “Bridging thevocabulary gap between health seekers and healthcare knowledge,”IEEETrans. Knowl. Data Eng., vol. 27, no. 2, pp. 396–409, Feb. 2015.

[36] F. A. Pozzi, D. Maccagnola, E. Fersini, and E. Messina, “Enhanceuser-level sentiment analysis on microblogs with approvalrelations,” in Proc. 13th Int. Conf. AI* IA: Advances Artif. Intell.,2013, pp. 133–144.

[37] R. Neumann and F. Strack, “mood contagion”: The automatictransfer of mood between persons,” J. Personality Social Psychology,vol. 79, pp. 211–223, 2000.

[38] C. Tan, L. Lee, J. Tang, L. Jiang,M. Zhou, and P. Li, “User-level sen-timent analysis incorporating social networks,” in Proc. SIGKDDInt. Conf. Knowl. Discovery DataMining, 2011, pp. 1397–1405.

[39] Wenbin Tang, Honglei Zhuang, and Jie Tang, “Learning to infersocial ties in large networks,” in Proc. Eur. Conf. Mach. Learn.Knowl. Discovery Databases, 2011, pp. 381–397.

[40] Y. R. Tausczik and J. W. Pennebaker, “The psychological meaningof words: Liwc and computerized text analysis methods,” J. Lan-guage Social Psychology, vol. 29, no. 1, pp. 24–54, 2010.

[41] M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas,“Sentiment strength detection in short informal text,” J. Amer. Soc.Inform. Sci. Technol., vol. 61, no. 12, pp. 2544–2558, 2010.

[42] Svetnik V, “Random forest: A classification and regression tool forcompound classification and qsar modeling,” J. Chemical Inform.Comput. Sci., vol. 43, no. 6, pp. 1947–1958, 2003.

[43] B. Verhoeven, W. Daelemans, and B. Plank, “Twisty: A multilin-gual twitter stylometry corpus for gender and personalityprofiling,” in Proc. 10th Int. Conf. Language Resources Eval., 2016,pp. 1632–1637.

[44] C. Wang, J. Tang, J. Sun, and J. Han, “Dynamic social influenceanalysis through time-dependent factor graphs,” Proc. Int. Conf.Advances Social Netw. Anal. Mining, 2011, pp. 239–246.

[45] X. Wang, J. Jia, P. Hu, S. Wu, J. Tang, and L. Cai, “Understandingthe emotional impact of images,” in Proc. 20th ACM Int. Conf.Multimedia, 2012, pp. 1369–1370.


[46] L. Xie and X. He, “Picture tags and world knowledge: Learningtag relations from visual semantic sources,” in Proc. 21st ACMMultimedia Conf., 2013, pp. 967–976.

[47] Y. Xue, et al., “Detecting adolescent psychological pressures frommicro-blog,” Proc. Int. Conf. Health Inform. Sci. Springer Int.Publishing, 2014, pp. 83–94.

[48] J. B. Yang, M. N. Nguyen, P. P. San, X. L. Li, and S. Krishnaswamy,“Deep convolutional neural networks on multichannel time seriesfor human activity recognition,” in Proc. Int. Joint Conf. Artif. Intell.,2015, pp. 3995–4001.

[49] Y. Yang, et al., “How do your friends on social media discloseyour emotions?” in Proc. 28th AAAI Conf. Artifi. Intell., 2011,pp. 306–312.

[50] S. Zeng, M. Lin, and H. Chen, “Dynamic user-level affect analysisin social media: Modeling violence in the dark web,” in Proc. IEEEInt. Conf. Intell. Secur. Informat., 2011, pp. 1–6.

[51] Q. Zhang and B. Goncalves, “Topical differences between chineselanguage Twitter and sina weibo,” Comput. Sci., 2015.

[52] Y. Zhang, J. Tang, J. Sun, Y. Chen, and J. Rao, “Moodcast: Emotionprediction via dynamic continuous factor graph model,” Proc.IEEE 13th Int. Conf. Data Mining, 2010, pp. 1193–1198.

[53] J. Zhao, L. Dong, J. Wu, and K. Xu, “Moodlens: An emoticon-based sentiment analysis system for chinese Tweets,” in Proc. 18thACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2012,pp. 1528–1531.

Huijie Lin is currently working toward the PhD degree at Tsinghua Uni-versity. His research interests include social media analysis and affectivecomputing.

Jia Jia received the bachelor’s degree from Tsinghua University, in 2003,and received the PhD degree from Tsinghua University, in 2008. She isan associate professor in the Department of Computer Science andTechnology, Tsinghua University. Her main research interest is socialaffective computing and human computer speech interaction. She isserving as the secretary-general of the Professional Committee ofSpeech in Chinese Information Processing Society, and also a committeemember of the Multimedia Federation in China Society of Image andGraphic. She has been awarded the ACM Multimedia Grand ChallengePrize (2012) and the Scientific Progress Prizes from the National Ministryof Education (2009 and 2016).

Jiezhong Qiu is currently working toward the PhD degree at TsinghuaUniversity.

Yongfeng Zhang is a postdoc research associate with the University ofMassachusetts Amherst, USA.

Guangyao Shen is currently working toward the master’s degree atTsinghua University.

Lexing Xie received the BS degree from Tsinghua University, Beijing,China, in 2000, and the MS and PhD degrees from Columbia University,in 2002 and 2005, respectively, all in electrical engineering. She is a lec-turer in the Research School of Computer Science at the AustralianNational University. She was with the IBM T.J. Watson Research Center,Hawthorne, New York from 2005 to 2010. Her recent research interestsare in multimedia mining, machine learning and social media analysis.She has won several awards: the best conference paper award in IEEESOLI 2011 and the best student paper awards at JCDL 2007, ICIP2004, ACMMultimedia 2005, and ACM Multimedia 2002.

Jie Tang is an associate professor with the Department of ComputerScience and Technology, Tsinghua University. His main research inter-ests include data mining algorithms and social network theories. He hasbeen a visiting scholar with Cornell University, Chinese University ofHong Kong, Hong Kong University of Science and Technology, andLeuven University. He has published more then 100 research papers inmajor international journals and conferences including: KDD, IJCAI,AAAI, ICML, WWW, SIGIR, SIGMOD, ACL, Machine Learning Journal,TKDD, and TKDE.

Ling Feng is a professor of computer science and technology withTsinghua University, Beijing. Her research interests include context-aware data management toward ambient intelligence, knowledge-basedinformation systems, data mining and warehousing, and distributedobject-oriented database management systems. She has publishedmore than 150 scientific articles in high-quality international conferencesor journals, and received the 2004 Innovational VIDI Award by the Neth-erlands Organization for Scientific Research, the 2006 Chinese Chang-Jiang Professorship Award by the Ministry of Education, and the 2006Tsinghua Hundred-Talents Award.

Tat-Seng Chua joined the National University of Singapore, Singapore,in 1983, and spent three years as a research staff member with the Insti-tute of Systems Science, National University of Singapore. He was theacting and founding dean of the School of Computing, NationalUniversity of Singapore, from 1998 to 2000. He is currently the KITHCTchair professor with the School of Computing, National University of Sin-gapore. His research interests include multimedia information retrieval,multimedia question answering, and the analysis and structuring ofuser-generated contents. He has organized and served as a programcommittee member of numerous international conferences in the areasof computer graphics, multimedia, and text processing. He was the con-ference co-chair of ACM Multimedia in 2005, the Conference on Imageand Video Retrieval in 2005 and ACM SIGIR in 2008, and the technicalPC co-chair of SIGIR in 2010. He serves on the editorial boards of theACM Transactions of Information Systems, Foundation and Trends inInformation Retrieval, The Visual Computer, and the Multimedia Toolsand Applications. He is on the steering committees of the InternationalConference on Multimedia Retrieval, Computer Graphics International,and Multimedia Modeling Conference Series. He serves as a member ofinternational review panels of two large-scale research projects inEurope. He is the Independent director of two listed companies inSingapore.

" For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


Date post:	20-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

1820 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA...

Documents