Measuring Mobile Phone Use: Self-Report VersusLog Data
Jeffrey Boase
Ryerson University, 350 Victoria Street, Toronto, Ontario M5B 2K3
Rich Ling
IT University of Copenhagen and Telenor, Rued Langgaards vej 7, 2300 Copenhagen, Denmark
Approximately 40% of mobile phone use studies published in scholarly communication journals basetheir findings on self-report data about how frequently respondents use their mobile phones. Usinga subset of a larger representative sample we examine the validity of this type of self-report data bycomparing it to server log data. The self-report data correlate only moderately with the server log data,indicating low criterion validity. The categorical self-report measure asking respondents to estimate‘‘how often’’ they use their mobile phones fared better than the continuous self-report measure askingthem to estimate their mobile phone activity ‘‘yesterday.’’ A multivariate exploratory analysis furthersuggests that it may be difficult to identify under- and overreporting using demographic variablesalone.
Key words: measurement, mobile phones, self-report, log data
doi:10.1111/jcc4.12021
Introduction
Mobile phones are without question the most common form of electronically mediated communicationin the world. We have adopted the mobile phone as a safety link, and we have adopted it to facilitatecoordination in the family and at work. Textile weavers in Nigeria (Jagun, Heeks, & Whalley, 2008)fishers in India (Jensen, 2007), and lovers in Mozambique (Archambault, 2011) have all adopted themobile phone for their various purposes. Teens in the US, as well as many other countries, use themobile phone to send and receive an endless number of texts on all the large and small events of theirlives (Lenhart, et al., 2010). Given these diverse uses, it is a fertile area of research. Understanding howwe use the mobile phone helps to understand the functioning of society.
The ability to do research on mobile telephony (or for that matter any form of mediatedcommunication) however is based on the ability to use valid data. In many cases research on the mobilephone relies on questionnaires. In the questionnaires, respondents are often asked to estimate thefrequency or the duration of their mobile phone use either in summary form or during a specific timeperiod. Alternatively respondents can be asked to use a diary or some other device to quantify their use.These methods are nearly the same methods used to collect data on internet and social media use.
There are clearly problems with these approaches. It is difficult for people to remember or tocharacterize their use. In addition there may be different motivations to over- or underreport mobile
phone use. For people who see use as a marker of their popularity, there will be the impulse to padthe numbers. The opposite is true for people with a more sober relationship to technology and whofeel the device is not so central to their lives but who nonetheless have one. We are interested to get ahandle on the reliability of this information. In this paper we compare self-report measures of mobilephone voice and SMS (Short Message System; known as ‘‘texting’’ or ‘‘text message’’) use with serverlog data to evaluate the criterion validity of self-report measures. We further conduct an exploratoryanalysis to examine the extent to which demographic characteristics are associated with under- andoverreporting in self-report measures. Our analysis has two main implications. First, given that manyinfluential studies of mobile communication rely on self-report measures of mobile phone use, ourevaluation of these measures has strong implications for the field of mobile research as well as studieson other forms of mediated communication that rely on similar methods. Second, given that mostresearch involving self-report measures also collects demographic data, our exploratory analysis of thedemographic characteristics associated with under- and overreporting of mobile phone usage informsexisting and future research about the types of respondents that are likely to contribute to the decreasedvalidity of self-report measures.
Self-Report Measures in Mobile Communication Studies
To confirm that the self-report measures we examine in this paper are indeed commonly used inscholarly communication research we conducted an analysis of journal articles found through the SocialSciences Citation Index. Using the ‘‘communication’’ filter to focus only on publications appearingin communication sources, we searched with the keywords ‘‘mobile phone.’’ We then manually siftedthrough the results and removed papers that were not based on empirical research, leaving us with 41journal papers published between 2003 and 2010.
In many cases the authors of these papers only mention the general nature of the questions askedand do not provide their exact wording. However, enough description was provided for us to identifytwo common types of self-report measures that operationalize intensity of mobile phone use: frequencyand duration. Frequency measures typically asked respondents to report on the number of calls or SMSmessages sent and received in a single day, or how often they use their mobile phones in more generalterms (e.g. ‘‘less than once a month,’’ ‘‘one to three times a month,’’ ‘‘once a week,’’ etc.). Durationquestions ask respondents to estimate amount of time that they spend using mobile phones in a singleday or over several days. Duration studies often involve the use of time diaries where respondents writedown the amount of time that they spend communicating using mobile phones. Sometimes durationstudies simply ask respondents to estimate the amount of time that they spend using their mobilephones during a short period of time. Other types of self-report mobile phone use measures include thenumber of friends and family contacted regularly by mobile phones, the places where mobile phonesare used, and self-perception attitudes about how extensively mobile phones are used. Approximately40% of papers reviewed include at least one frequency of use measure, 27% include at least one durationof use measure, 9% rely on self-report measures other than frequency or duration to gauge level ofuse, and the remaining papers do not include any quantitative mobile phone use measures. Correlationanalysis showed little association between the type of measure used and the number of times that apaper has been cited.
The variety of approaches employed to measure mobile phone use indicates that there is not a singlecommonly accepted approach to gathering data on this issue, likely because different questionnairesare developed for different purposes. Nevertheless, the fact that approximately 40% of the papersreviewed include at least one self-report measure regarding frequency of mobile phone use indicates
that collecting this type of information is applicable to a variety of studies. This is not surprising giventhat frequency of mobile phone use is indicative of the extent to which mobile phones have becomeembedded into everyday life. Moreover, as with any study in mediated communication, establishingthat a technology is used at least to some extent is necessary before further implications of its use canbe investigated. Given the wide applicability and popularity of self-report frequency of use measures wefocus specifically on this type of measure in this study.
Examining Self-Report Measures of Mobile Phone Use
Although communication researchers often use self-report measures of mobile phone use, we couldidentify only two studies that examined the validity of these measures. One is a convenience samplestudy of 93 volunteers conducted by Parslow, Hepworth, and McKinney (2003). In this study volunteerswere asked to report on the number of incoming and outgoing voice calls during the previous day,week, or month, and further estimate the number of minutes spent on the voice calls. Their responseswere then compared to log data collected from four different mobile phone operators. Their resultsshowed moderate correlation between the self-report and log data, indicating that self-report measuresdid not fully represent actual use patterns. There was also evidence of over-reporting, especially amongusers with relatively low levels of activity. In another study, Cohen, and Lemish (2003) used a quotasample to compare self-report measures with real time data collection using Interactive Voice Responsetechnology over a 5-day period. They found the correlation between the questionnaire data and billingdata regarding the frequency of calling to be moderately strong, indicating a reasonable fit between theself-report measures and actual behavior.
While these two studies represent an important first step towards understanding the validity ofself-report measures in mobile communication studies, the nonrandom nature of their samplingmethods warrants further study of this issue. Moreover, these studies focused only on voice callingactivity and did not include measures of SMS messaging.
In related research, social network studies have looked at self-report measures of communicationnetworks. Using four different samples, Bernard and Killworth (1977), conclude that respondentsdo not know with any accuracy the individuals with whom they communicate. In another paperthat examines the results from seven experiments, Bernard, Killworth, and Sailer (1982), argue thatself-report of communication behavior bears no useful resemblance to actual communication behavior.Freeman, Romney, and Freeman (1987) also find low levels of accuracy in respondents’ ability toidentify individuals who coattended colloquium presentations with them, further finding that errorswere biased by long-term attendance patterns. Although none of these studies examine self-report inregards to mobile phone use, they do indicate that problems with self-report measures extend beyondmobile phone use as a particular form of communication.
There are three main reasons to question the validity of self-report measures regarding mobilephone use: cognitive burden, social desirability, and conceptual validity. Cognitive burden occurs whenrespondents attempt to recall behaviors that they do not typically think about or record on a regularbasis. To decrease this burden respondents often use heuristic shortcuts that decrease the validityof measures (Sudman, Bradburn, & Schwartz, 1996). Moreover, since mobile phone interaction ispotentially regarded as a more general indicator of sociability, interest in increasing desirability mayoccur and cause respondents to overreport the extent to which they use their mobile phones. Finally,even if self-report measures of mobile phone activity are accurate, there is some reason to question theextent to which they measure the concepts that they are intending to measure. In particular, questionsthat focus on mobile phone activity in a single day are suspect because it is unclear if the activity
occurring in that day is actually representative of more general levels of activity. For individuals withirregular schedules, day-to-day mobile phone activity may fluctuate dramatically.
Method
In this study we draw on a combination of data collected through a nationally representative internet-based survey conducted in Norway during October and November of 2008 and sever log data for themonth in which respondents completed the survey. In order to ensure the representativeness of thesample, participants were recruited from an earlier telephone survey and a few questionnaires were givenover the phone to older respondents who did not have internet access. Two versions of the mobile usequestion were included in the survey: a frequency of mobile activity occurring ‘‘yesterday’’ and frequencyof mobile phone use occurring over longer periods of time, e.g. on a weekly basis. The survey data thatwe use in this study is drawn from the second wave of a two-wave study. In total, 1499 respondentsagreed to participate in the second wave (65% response rate), and respondents showing inconsistenciesbetween the first and second wave (for example, radically different ages, change in gender, etc. betweenwave 1 and wave 2) were dropped leaving a total of 1382 respondents. Researchers potentially had accessto the log data for 613 of these second-wave respondents because these respondents were customersof the telecommunication company funding the survey. Four hundred twenty-six of those customersgranted their informed consent for the researchers to access their log data.
The demographic composition of those granting access to their data is as follows: 50% are female,58% are married, 23% hold a college degree, 58% are working full time, the mean age is 49 years (SD =15.00; skewness = −.17), the mean household size is 2.7 individuals (SD - 1.41; skewness - .85). Thosegranting access to their log data were similar with the other second wave respondents who did not grantaccess to their log data in terms of their sex, household size, and education. However, compared to theother respondents, those who granted access to their log data were a mean of approximately 8 yearsolder, more likely to be married (58% married versus 42% married), and more likely to be workingfull time (58% working full time versus 48% working full time). A separate analysis of second-waverespondents indicates that the subset of respondents used in the analysis reported here–i.e. customerswho granted access to their log data–differ from other respondents in our representative sample mainlybecause they are customers of a particular telecommunication company and not because of self-selectionin regards to their willingness to share their log data. This bodes well for the generalizability of ourresults, although we caution that these results are still somewhat limited insofar as they apply mostdirectly to individuals who share the demographic traits of our subsample.
Self-Report Measures
The telephone survey data included two types of self-report measures aimed at measuring intensity ofmobile phone use for both voice and SMS messages. The first type of measure is continuous in natureand asked respondents to report the number of times that they used their mobile phone ‘‘yesterday’’ foroutgoing voice calls and SMS messages. In one question they were asked: ‘‘How many times did you usea mobile phone yesterday to call others, either for work or privately.’’ In another question respondentswere asked, ‘‘How many text messages did you send yesterday, either for work or privately?’’ We referto these types of continuous self-report questions as ‘‘yesterday’’ measures.
The second type of question is categorical in nature and required respondents to report on howoften they use their mobile phones by choosing the most appropriate response from a set list. In onequestion they were asked, ‘‘How often do you use a mobile phone to call others (not texting or SMS)?
More than 10 times a day, 5–10 times a day, 2–4 times a day, at least once a day, 3–6 times a week,1–2 times a week, Less often, Never.’’ In another question they were asked, ‘‘How often do you usethe mobile phone to send or receive text messages/SMS?’’ and were given the same set of categoricalresponse options. We refer to these types of categorical self-report questions as ‘‘how often’’ measures. Notethat all self-report measures except for this last categorical measure of SMS messages refer only tooutgoing activity and that all questions have been directly translated from Norwegian.
Log Data
Server log data includes all outgoing voice and SMS events captured during the month in which thesurvey took place. Three types of variables are constructed with this data for both voice and SMS events.The first is simply the number of outgoing events that month. The second is the average number ofoutgoing events per day as calculated by dividing the total number of events by the number of days inthat month. This type of variable is used for comparison with the yesterday measures of daily use. Thethird is categorical, constructed using the same categories as those used in the how often measures, i.e.More than 10 times a day, 5–10 times a day, 2–4 times a day, at least once a day, 3–6 times a week, 1–2times a week, Less often, Never. Here a calculation is used to select the most appropriate category fromthese choices using the log data.
Conceptualization and Operationalization
To allow for a fair comparison between the self-report and log measures it is important to be clear aboutexactly what is being compared. Our comparison is concerning the various possible operationalizationsof a concept, rather than the accuracy of the measures themselves. In other words, we are more interestedin the extent to which self-report measures can adequately operationalize a particular concept than theextent to which they directly measure the exact nuances of the behavior about which they are asking.For example, we are more interested to know if asking people to report on the number of people theycalled yesterday is a good operationalization of mobile phone use, in general, than if respondents areable to recall this behavior on a specific day with a high degree of accuracy.
The broad concept that we are focusing on is intensity of mobile phone use, and the specificdimension of this concept being examined is frequency of mobile phone use. Among the variousways of operationalizing this dimension, monthly server log data represents one of the best possibleoperationalizations because it is highly accurate. Indeed, it must be accurate because it is the basisof billing. Moreover, since the data is collected for a sustained period of time it is less affected byunusual events. Although it may be the case that monthly server log data collected during the holidayseason may be unusually high, the data that we use in this analysis is not collected during thisperiod of time. Moreover, all the respondents included in our analysis indicated that they are the soleowner of their mobile phone and that they do not use anyone else’s mobile phone. In addition, thepractice of ‘‘SIM-switching’’ that is often seen in developing countries is not common in Norway.1
Thus, for all intents and purposes, the log data represents the total mobile phone use of a singleindividual.
Given that the server log data represents an ideal way of operationalizing frequency of use, ouranalysis focuses on comparing it to the two types of operationalizations that are less ideal–namely,the two types of self-report measures discussed above. While these measures are clearly aimed atoperationalizing frequency of use, issues of respondent recall, social desirability, and generalizabilitygive reason to doubt their validity. In the analysis that follows we examine the criterion validity of the
self-report measures by comparing them to the more ideal sever log measures to address the followingresearch questions.
Research Question 1: How well do the self-report measures compare to sever log data?Since it is likely that at least some respondents will be more prone to over- or underreport their
usage and identifying such respondents may help researchers to develop better measures and decreasetheir negative impact in the analysis, we further ask:
Research Question 2: What are the demographic characteristics associated with over- and underre-porting of mobile phone use?
We focus on demographic characteristics primarily because such characteristics are typicallycollected in self-report surveys.
Analysis and Results
To address our first research question, we start by comparing our log data with the self-report measuresusing pairwise correlations. The results show that although the self-report measures are significantlycorrelated with the log data (p < .001 for all correlations), the correlations are not as strong as could beexpected from measures that operationalize the same concept equally well.
The correlations between the yesterday self-report questions and the voice log measures are .55, p< .001, for outgoing voice calls, and .58, p < .001, for outgoing SMS messages. In order to examine thepossibility that the presence of outliers adversely affected these correlations, we preformed logarithmictransformations on these variables. Our results show lower correlations with these transformed variables,at .23, p < .001, for outgoing voice calls and .41, p < .001, for outgoing SMS messages.
The correlations between the how often self-report questions and the SMS message log data are .48,p < .001, for voice calls and .35, p < .001, for SMS messages. Correlations with a transformed logvariable are .66, p < .001, for voice calls and .74, p < .001, for SMS messages. Given that the transformedvariables better controlled for the influence of outliers, these results indicate that the correlationsbetween self-report and log data for the how often measures were stronger than the correlations for theyesterday measures.
In sum, although these correlations may be high for variables that measure two different yetassociated things–for example, mobile phone use and life satisfaction–they are not high for variablesthat are supposed to be measuring exactly the same thing. Overall, correlating self-report measures tolog data generally indicates that the self-report measures are only somewhat similar to the superiorlog measures. The most promising self-report measure was the categorical how often measure of SMSmessaging. We note that this finding is unexpected given that this measure focused on both outgoingand incoming messages, while the log data captured only outgoing SMS messages, though there is oftensymmetry between in and out-going calls. The other self-report measures that show less correlationwith the log data asked about outgoing voice and SMS events only, making them theoretically morecomparable to the outgoing log data.
Next, we compare the self-report measures directly to the log data to understand exactly how theself-report measures differ from the log data, and to confirm that these differences are indeed significant.For this set of analyses we use log variables that were recoded to make them directly comparable tothe self-report measures. We start by describing the difference between yesterday self-report and logmeasures, as presented in Table 1. We find that respondents report making more voice calls yesterdaythan the log data shows that they make during an average day. The mean number of reported callsyesterday is 3.88, while the mean number of calls logged on an average day is 40 percent lower thanthat number (2.38). The median number of calls reported yesterday is 3, while the median number of
calls logged on an average day is less than half that number (1.43). We also find a much larger standarddeviation for the self-report yesterday measures than the voice log measure (4.8 versus 2.99, respectively),indicating a much greater range of variability in the responses given through this self-report measurethan we would expect based on the voice log measure. As with the voice self-report yesterday measures,the SMS message self-report measures were also higher than the log SMS measures. Respondents reporta mean of 6.19 SMS messages yesterday, while the mean number of SMS messages logged in an averageday is 3.95. The median number of self-report SMS messages is 3.00, while the median number oflogged SMS messages is 1.87. The standard deviation for the self-report SMS measure is 13.98, whilethe standard deviation of the log SMS measure is 8.81.
A one-tail t test comparing the self-report and log measures of voice calls occurring yesterdayconfirms that the mean of the self-report measure is significantly higher than the mean of the logmeasure, t(425) = 7.72, p < .001. A one-tail t test comparing the self-report and log measures for thenumber of SMS messages sent yesterday also shows that mean of the self-report measure is significantlyhigher than the mean of the log measure, t(425) = 4.07, p < .001.
We also compared the how often self-report measure to recoded log variables. Table 2 shows thepercentage of respondents in each category of these self-report and log measures. The results generallyshow that a greater percent of respondents in each category of the self-report measures than in eachcategory of the log measures. The main exception is that respondents whose log data show an extremelylow amount of usage do not tend to report this low usage in the self-report measures. The log datashowed that 17% of the respondents made less than one voice call per week, however only 3% of the
Table 2 Percent comparison of the ‘‘how often’’ self-report measure and categorized call log activity
11+ times a day 9 3 6 11 7 45-10 times a day 15 10 5 24 15 92-4 times a day 32 28 4 28 27 1At least once a day 14 18 −3 10 13 −33-6 times a week 19 15 3 14 10 41-2 times a week 8 9 −1 8 6 1Less often 3 17 −14 5 23 −18Total percent 100 100 100 100
respondents self-reported having this low level of calling activity. This finding may point to respondents’reluctance to report low levels of socialization in the questionnaire. The log data also showed that23% of the respondents sent less than one text message per week, yet only 5% of the respondentsself-reported exchanging less than one text message per week.
To confirm that the self-report and log measures for the categorical how often variables differedsignificantly according to more stringent statistical criteria, we conducted a Chi-square goodness offit tests using csgof program installed as an add-on command in STATA. Using this program wewere able to compare the frequency of respondents in each category of the self-report measures withthe frequencies that we would expect based on the log data. The results of this test showed that thefrequencies of respondents in the categories of the voice self-report measure differed significantly fromthe frequencies that could be expected based on the voice log measure, chisq(6) = 121.49, p = 0.Similarly, results showed that the frequencies of respondents in the categories of the SMS self-reportmeasure differed significantly from the frequencies that could be expected based on the SMS logmeasure, chisq(6) = 124.71, p = 0.
In sum, regarding our first research question, our results show that self-report measures of voicecalling and SMS messaging generally do not compare favorably to log measures. This is true for boththe self-report measures that require respondents to report on these behaviors yesterday, and for theself-report measures that requires respondents to report how often these behaviors occur by selectingfrom a number of predetermined categories. In general, respondents are more likely to overreport thanto underreport voice calling and SMS messaging.
To address our second research question, we conducted an exploratory multivariate analysis toexamine the extent to which demographic characteristics can be used to predict under- and overreportingof voice calling and SMS messaging. We control for whether respondents pay their mobile phone billscompletely on their own since this likely affects the self-monitoring behavior that would assist in recalland is also related to factors such as employment status, age, and education.
We first examine under- and overreporting of the yesterday measures. Accordingly, two independentvariables are used: one indicating the amount of overreporting that occurs when the self-report measureis greater than log measure of an average day, and the other indicating the opposite. The results of thisanalysis are presented in Table 3.
In regard to voice calling, different demographic traits are associated with overreporting thanare associated with underreporting. Being a male is positively associated with overreporting whilehousehold size is positively associated with underreporting (p < .001 for both coefficients). Workingfull or part time is positively associated with underreporting (p < .05). In regard to SMS messaging, ageis negatively associated with both overreporting and underreporting (p < .001 for both coefficients),implying that young adults provide less valid reports than those who are older. Being a female and beingemployed are both positively associated with underreporting of SMS messages (p < .05). Being marriedis negatively associated with overreporting of SMS messages (p < .01). The adjusted R2 statistics forthese models are all relatively low, indicating that demographic traits do not explain a great amount ofthe variance in under- and overreporting.
We also used a multivariate analysis to explore demographic factors associated with under- andoverreporting that occurred with the how often self-report variable, using dichotomous variables toindicate under- and overreporting that occurred when respondents selected a category that differedfrom the appropriate category indicated by the log data. The results of this analysis are presented inTable 4.
This multivariate analysis shows that only a few factors are significantly associated with under- andoverreporting. In regards to voice calls, only being a male was significantly and negatively associated withunderreporting (p < .05). In regard to SMS messaging, only being a male and age were significantly and
positively associated with overreporting (p < .05). That is, this analysis indicates that males significantlyoverreport their voice mobile use. Further, males and also people who are older overreport their use ofSMS. As with the multivariate analysis of the yesterday self-report measures, the low Pseudo R2 statisticsof these multivariate models indicates that demographic traits do not explain a great amount of thevariance in under- and overreporting that occurs with the how often self-report measures.
In sum, our exploratory analysis links a few demographic traits to under- and overreporting,especially with regard to the yesterday self-report measure. Nevertheless, these demographic traits tonot explain a large amount of variance in our models, which suggests that it may be difficult to identifyand compensate for under- and overreporting using demographic traits alone.
This study faces a few limitations. First, given that it takes place in Norway, it is possible that the resultsmay not generalize well to other cultures or contexts. While this problem is common to most studiesand there is no obvious reason to think that Norwegians would be more likely to under- or overreporttheir mobile phone usage than people in other countries, this point is worth bearing in mind. Second,our focus has been on how well these measures operationalize the concept of frequency of mobilephone use, rather than the specific accuracy of these measures in their own right. For example, we aremore interested in how well questions about mobile activity that occurred yesterday generally measurefrequency of mobile phone use than we are interested in how well they accurately reflect activity thatoccurred on that specific day. For this reason these results do not apply to studies for which it isimportant to have an accurate measure of activity occurring on a specific day. This is not to say that these‘‘yesterday’’ measures are necessary accurate–just that we cannot comment on their accuracy basedon these findings. Third, although our data comes from a subset of a nationally representative sample,it is possible that groups who systematically avoid participating in these surveys answer self-reportquestions differently than those who participate. However, we find that the demographic compositionof the subsample is mostly accounted for by the fact that respondents in the subsample were customersof the telecommunications company funding this study and customers of this company tended to havethese demographic traits. Another issue is that this study is founded on behavioral data in additionto a few demographic variables. The data that was available did not include eventual motivationalquestions because the questionnaire was focused on other issues. This does not mean that there is notan interesting link between motivational variables and actual traffic data, only that it was not a partof this study.
Finally, although all respondents who participated in this study report that they are the sole usersof the phones for which we collected the log data, it is possible that family members or friends mayoccasionally borrow their phones thereby decreasing the validity of the log data. We have little reasonto believe that this is a common problem in our data, however, such an error may have occurred insome cases.
Discussion and Conclusion
Our results indicate that while self-report indicators aimed at measuring the frequency of mobilephone use are not completely out of line with actual behavior, our analysis shows that these self-report measures suffer from low criterion validity. That is to say, they are generally suboptimal waysof operationalizing this dimension of mobile phone use. These measures only moderately correlatewith actual behavior, they vary more widely than actual behavior, and they are prone to overre-porting. The categorical how often frequency measure that used predetermined response categoriesfared better than the continuous yesterday frequency measure that used an open-ended responseapproach. Perhaps this was because providing predetermined responses constrained respondentswho would have more widely overreported, and cued them to an appropriate range of possi-ble responses. The downside of using the categorical measure is that it can be more difficult toincorporate it into certain types of analyses and it provides less information than the continuousmeasure.
The results of this study have strong implications for the many studies that use these types ofmeasures. As discussed in our analysis of the mobile phone literature, approximately 40 percentof mobile phone use studies published in scholarly communication journals base their findingson self-report frequency measures. Following from the notion of W.I. Thomas, our perception of
use may be real in its consequences. That is, if someone thinks that they are a heavy or a lightuser of mobile communication they will likely arrange other contingent activities based on thatperception.
That said, given the moderate correlations between these self-report measures and log data thereis good reason to be suspicious of studies that find significant correlations between these self-reportmeasures and other variables. Moreover, the implications for future research are potentially seriouswhen considering access to more accurate trace data–such as mobile phone log or server log data–isgenerally difficult, time consuming, and expensive to obtain. Researchers might improve self-reportmeasures by asking respondents to access the voice and SMS logs of their mobile phones or find oldphone bills during interviews or while completing paper or web questionnaires. We caution that, thisapproach may result in frustration and additional time to complete surveys as respondents may noteasily be able to pull this information from their mobile phones or locate old phone bills. The use ofdiaries is also a possibility, though this also implies a greater commitment by respondents and moredifficulty in gathering data. It is also possible to consider a type of proactive data collection system suchas calling people at random times via their mobile phones and asking for the previous mobile-basedinteraction. Another approach would be to use mobile applications designed to pull log data frommobile phones and send it directly to researchers. For a detailed discussion of the merits of such anapproach, see Raento, Oulasvirta, and Eagle (2009).
To inform current and future studies, we ran exploratory multivariate analyses to examinethe possible correlation between demographic traits and under- and overreporting. We focused ondemographic traits because they are generally present in most surveys and therefore are obvious targetsfor identifying and compensating for inaccurate self-reporting behavior. While we did find someassociations between demographic traits and misreporting on the yesterday measures, we didn’t findmany associations with the categorical how often measure. Moreover, the low R2 statistics in our modelsindicate that demographic traits will not go far in helping to identify and compensate for under- andoverreporting in both the yesterday and the how often measures.
In conclusion, our results point towards the need to improve self-report measures that operationalizefrequency of mobile phone use. Given that these measures are often used in influential mobile phonestudies, more resources should be dedicated to understanding the sources of their weakness andexamining alternative measures that operationalize the same concept and can be easily deployed intosurvey questionnaires.
There are always boundaries in terms of funding, time, the constraints of the questionnaire method,etc. As a result we often muck along as best we can and we accept the results of others given theconstraints that they work under. This ignores, however the larger question of the validity of the results.In a perfect researcher’s world there would be access to valid and reliable data that would not threatenthe privacy of the individual. However, we do not live in that world. We have to think about howto get the best data given the instruments and the constraints that we currently have and we have tothink about better data collection methods. Finally, we need to think about the way that our resultsare reported and the caution needed in pushing the meaning of the results beyond what the data cansupport.
Note
1 An individual who practices SIM switching will have SIM cards from several different operators andwill switch between these during the day in order to take advantage of daily tariff differencesbetween the operators
Archambault, J. S. (2011). Breaking up because of the phone and the transformative potential ofinformation in Southern Mozambique. New Media & Society, 13(3), 444–456.
Bernard, H. B., & Killworth, P. D. (1977). Informant accuracy in social network data II. HumanCommunication Research, 4(1), 3–18.
Bernard, H. B., Killworth, P. D., Sailer, L. (1982). Informant accuracy in social network data: Anexperimental attempt to predict actual communication from recall data. Social Science Research,11(1), 30–66.
Cohen, A. A., & Lemish, D. (2003). Real time and recall measures of mobile phone use: Somemethodological concerns and empirical applications. New Media & Society, 5(2), 167–183.
Freeman, L. C., Romney, A. K., Freeman, S. C. (1987). Cognitive structure and informant accuracy.American Anthropologist, 89(2), 310–325.
Jagun, A., R. Heeks, and J. Whalley. (2008). The impact of mobile telephony on developing countrymicro-enterprise: A Nigerian case study. Information Technologies and International Development,4(4), 47–65.
Jensen, R. (2007). The digital provide: Information technology, market performance and welfare in theSouth Indian fisheries sector. The Quarterly Journal of Economics, 122(3), 879–924.
Lenhart, A., et al. (2010). Teens and mobile phones. Washington, DC: Pew Research Center.Parslow, R. C., Hepworth, S.J., & McKinneym P.A. (2003). Recall of past use of mobile phone headsets.
Radiation Protection Dosimetry, 106(3), 233–24.Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for social scientists.
Sociological Methods Research, 37(3), 426–454.Sudman, S., Bradburn, N. M., & Schwartz, N. (1996). Thinking about answers: The application of
cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.
About the Authors
Jeffrey Boase ([email protected]) is an assistant professor in the School of Professional Commu-nication at Ryerson University in Toronto, Canada. His research focuses on the relationship betweencommunication technology and social networks. Further information and publications can be foundat: http://www.ryerson.ca/∼jboase/
Address: School of Professional Communication, Ryerson University, 350 Victoria Street, Toronto,Ontario M5B 2K3
Rich Ling is a professor at IT University of Copenhagen in Copenhagen Denmark. His work has focusedon the social consequences of mobile communication.
Address: IT University of Rued Langgaards vej 7, 2300 Copenhagen
