NATIONAL CENTER FOR EDUCATION STATISTICS
Technical Report March 1997
National Household Education Survey
Feasibility of ConductingFollowup Surveys in theNational Household EducationSurvey
U.S. Department of EducationOffice of Educational Research and Improvement NCES 97-335
NATIONAL CENTER FOR EDUCATION STATISTICS
Technical Report March 1997
National Household Education Survey
Feasibility of ConductingFollowup Surveys in theNational Household EducationSurvey
J. Michael BrickMary CollinsElizabeth Davies
Westat, Inc.
Kathryn Chandler
National Center for Education Statistics
U.S. Department of EducationOffice of Educational Research and Improvement NCES 97-335
U.S. Department of EducationRichard W. RileySecretary
Office of Educational Research and ImprovementMarshall S. SmithAssistant Secretary
National Center for Education StatisticsPascal D. Forgione, Jr.Commissioner
The National Center for Education Statistics (NCES) is the primary federal entity forcollecting, analyzing, and reporting data related to education in the United States and othernations. It fulfills a congressional mandate to collect, collate, analyze, and report full andcomplete statistics on the condition of education in the United States; conduct and publishreports and specialized analyses of the meaning and significance of such statistics; assiststate and local education agencies in improving their statistical systems; and review andreport on education activities in foreign countries.
NCES activities are designed to address high priority education data needs; provideconsistent, reliable, complete, and accurate indicators of education status and trends; andreport timely, useful, and high quality data to the U.S. Department of Education, theCongress, the states, other education policymakers, practitioners, data users, and thegeneral public.
We strive to make our products available in a variety of formats and in language that isappropriate to a variety of audiences. You, as our customer, are the best judge of oursuccess in communicating information effectively. If you have any comments or suggestionsabout this or any other NCES product or report, we would like to hear from you. Pleasedirect your comments to:
National Center for Education StatisticsOffice of Educational Research and ImprovementU.S. Department of Education555 New Jersey Avenue, NWWashington, DC 20208-5574
March 1997
Suggested Citation
U.S. Department of Education. National Center for Education Statistics. Feasibility of Conducting Followup Surveys inthe National Household Education Survey , NCES 97-335, by J. Michael Brick, Mary Collins, Elizabeth Davies, andKathryn Chandler. Washington, DC: 1997.
Contact:Kathryn Chandler(202) 219-1767
iii
Table of ContentsPage
Background for the NHES................................ ................................ ................................ ................. 1
Rationale for a Longitudinal Feasibility Study................................ ................................ .................... 2
Design of the Followup Survey................................ ................................ ................................ ........... 3
Survey Procedures for the Followup................................ ................................ ................................ ... 4
Locating Procedures................................ ................................ ................................ .............. 4Schedule and Interviewing................................ ................................ ................................ ..... 6
Findings and Analysis................................ ................................ ................................ ........................ 7
Locating and Response Rates................................ ................................ ................................ . 7Response Status by Locating Characteristics ................................ ................................ .......... 7Response Status by Tracing Efforts ................................ ................................ ....................... 9Implications for Longitudinal Analysis ................................ ................................ ................... 11Longitudinal Response Rates ................................ ................................ ................................ . 15
Conclusions and Implications ................................ ................................ ................................ ............. 19
Cost Considerations................................ ................................ ................................ ............... 19Alternative Assessment of Change Over Time ................................ ................................ ........ 20
References ................................ ................................ ................................ ................................ ......... 22
List of Tables
Table 1. Number and percent of sampled persons, by response statusand address information status ................................ ................................ ........................... 8
Table 2. Number and percent of sampled persons, by response statusand telephone location ................................ ................................ ................................ ........ 8
Table 3. Number of sampled persons, by response status, telephone location,and address information ................................ ................................ ................................ ..... 9
Table 4. Number and percent of sampled persons, by response statusand tracing activity ................................ ................................ ................................ ............ 10
iv
List of Tables (continued)Page
Table 5. Number and percent of sampled persons, by response statusand all types of locating activities ................................ ................................ ....................... 12
Table 6. Followup response rates by child and household characteristics ................................ .......... 14
Table 7. Response rates from the 1988 Panel of the Survey of Incomeand Program Participation................................ ................................ ................................ .. 17
Table 8. Response and sample retention rates for the civiliansample of the National Longitudinal Survey of Youth: 1979 to 1991 ................................ .. 18
List of Figures and Exhibits
Figure 1. Process of locating and interviewing respondentsin the NHES:91 followup survey ................................ ................................ ........................ 5
Exhibit 1. Postcard mailed to NHES:91 followup sample................................ ................................ .... 6
1
Background of the NHES
Most large random-digit-dial telephone surveysconducted in the U.S. are cross-sectional studiesthat examine characteristics of the population at aparticular point in time. A longitudinal design isanother method for surveying the population thathas important advantages over repeated cross-sectional surveys for estimating changes over time.This report examines the feasibility of conductinga longitudinal survey when the base year data arefrom a random-digit-dial telephone survey. TheNational Household Education Survey was thevehicle for this research and is described belowprior to a more complete discussion of the rationalefor the study.
The National Household Education Survey(NHES) is a data collection system of the NationalCenter for Education Statistics (NCES), which hasas its legislative mission the collection andpublication of data on the condition of education inthe Nation. The NHES is specifically designed tosupport this mission by providing information onthose educational issues that are best addressed bycontacting households rather than schools or othereducational institutions. The NHES providesdescriptive data on the educational activities of theU.S. population and offers policymakers,researchers, and educators a variety of statistics onthe condition of education in the United States.
The NHES is a telephone survey of thenoninstitutionalized civilian population of the U.S.Households are selected for the survey usingrandom-digit-dialing (RDD) methods, and data arecollected using computer-assisted telephoneinterviewing (CATI) procedures. Approximately60,000 households are screened for eachadministration, and individuals within householdswho meet predetermined criteria are sampled formore detailed or extended interviews. The data areweighted to permit estimates of the entirepopulation. The NHES survey for a given yeartypically consists of a Screener, which collectshousehold composition and demographic data, andextended interviews on two substantivecomponents addressing education-related topics.In order to assess data item reliability and inform
future NHES surveys, each administration alsoincludes a subsample of respondents for areinterview.
The primary purpose of the NHES is to conductrepeated measurements of the same phenomena atdifferent points in time. Throughout its history,the NHES has collected data in ways that permitestimates to be tracked across time. This includesrepeating topical components on a rotating basis inorder to provide comparative data across surveyyears. In addition, each administration of theNHES has benefitted from experiences withprevious cycles, resulting in enhancements to thesurvey procedures and content. Thus, while thesurvey affords the opportunity for trackingphenomena across time, it is also dynamic inaddressing new issues and including conceptualand methodological refinements.
A new design feature of the NHES programimplemented in the NHES:96 was the collection ofdemographic and educational information onmembers of all screened households, rather thanjust those households potentially eligible for atopical component. In addition, this expandedscreening feature included a brief set of questionson an issue of interest to education programadministrators or policymakers. The totalScreener sample size was sufficient to producestate estimates of household characteristics for theNHES:96.
Full-scale implementations of the NHES have beenconducted in 1991, 1993, 1995, and 1996. Topicsaddressed by the NHES:91 were early childhoodeducation and adult education. The NHES:93collected information about school readiness andschool safety and discipline. The 1991components were repeated for the NHES:95,addressing early childhood program participationand adult education. Both components underwentsubstantial redesign to incorporate new issues anddevelop new measurement approaches. In theNHES:96, the topical components wereparent/family involvement in education and civicinvolvement. The NHES:96 expanded screeningfeature included a set of questions on public libraryuse.
2
In addition to its topical components, the NHESsystem has also included a number ofmethodological investigations. These have resultedin technical reports and working papers coveringdiverse topics such as telephone undercoveragebias, proxy reporting, and sampling methods. Thisseries of technical reports and working papersprovides valuable information on ways ofimproving the NHES.
Rationale for a LongitudinalFeasibility Study
The NHES employs a cross-sectional design,which allows for the collection of detailedinformation on independent samples of householdsthat are different across administrations. Incontrast, studies using a longitudinal design collectdata from a sample of respondents at more thanone point in time. The ability to track a sampleacross time, addressing the same researchquestions, can provide precise measurement of theprocess and outcomes of educational experiences.
Longitudinal studies are often undertaken to studyhow experiences and activities relate to outcomesthat occur later in time. For example, longitudinalstudies of students such as High School andBeyond (Zahs et al. 1995), the NationalEducational Longitudinal Study (Ingels 1994), andthe Beginning Postsecondary StudentsLongitudinal Study (Fitzgerald 1994) have beenused to measure relationships between theexperiences of students and later educationalachievement, enrollment in postsecondaryschooling, and career paths. For longitudinalstudies of young children, a primary objective is tobetter understand how early childhood experiences,including participation in educational programs,prepare children for success in school.
Longitudinal data collection programs also providea unique opportunity to examine the process ofchange. For example, with a longitudinalapproach to early childhood experiences, it ispossible to accurately capture the number andtypes of programs in which a child participates atdifferent points in his or her early years, as well as
to examine how the types of programs change inrelation to changing family composition andeconomic circumstances. Such data may be ofgreat value in assessing how these earlyexperiences are associated with later educationalprogress.
There are other methodological features associatedwith using longitudinal designs in educationalresearch (Bailar 1989). For example, cross-sectional surveys have attempted to assess changein children's development over time by collectingretrospective information from parents. However,reliance on respondents' memory can beparticularly error-prone. A longitudinal approachreduces the error associated with recall because thedata are collected near the time the events occur.The accuracy afforded by this approach maypermit the identification of relationships that mightbe obscured by the response errors inherent incross-sectional surveys.
Since one of the goals of the NHES is to monitorchanges over time, both nationally and for keysubgroups, longitudinal approaches have a naturalappeal. Duncan and Kalton (1987) describe avariety of measures of change and the designapproaches that are most advantageous for thesemeasures. A longitudinal design is the onlyapproach that permits estimates of changes at theindividual (micro) level. For example, alongitudinal design is necessary to examine howevents and experiences in a child's early life affectperformance later in life, assuming retrospectivedata collection is not deemed reliable. Theimplementation of a longitudinal design wouldgreatly enhance the analytic potential of the NHESdata sets. Assessing the ability of the NHES to bea vehicle to address these types of issues inaddition to providing cross-sectional estimates ofthe population was the motivation for this study.
While much is known about the difficulties andsuccesses associated with longitudinal studiesbased on institutional samples or in-person baseyear interviews, little is known about longitudinalstudies based on random-digit-dial (RDD)samples. Booth and Johnson (1985) report on astudy of just over 2,000 married couples who were
3
selected by RDD in the base year and wereinterviewed 3 years later, after interim trackingcontacts.
As a part of a methodological investigation, theNHES:91 contained an experiment to test thefeasibility of using the NHES as a mechanism toconduct longitudinal studies of young children.The complex design and methodologicalprocedures of the NHES present importantchallenges to undertaking a longitudinal approach.For example, random-digit-dial surveys typicallyhave lower response rates than personal interviewsurveys, with much of the loss in responseoccurring within the first moments of contact withthe household (Groves and Lyberg 1988). Iftracing RDD samples is also problematic,substantial additional numbers of cases could belost to the study. The net response rates that canbe obtained in a longitudinal study must beevaluated in light of this fact. Booth and Johnson(1985) for example, report a base year responserate of 65 percent and a followup completion rateof 78 percent, for a time 2 response rate of 51percent. The NHES feasibility study focuses onthe issue of response rates but also addresses otherchallenges associated with a longitudinal design.
The subsequent sections of the report describe thedesign, procedures, and results of the followupsurvey. The final section summarizes the researchand discusses other methods that may be used toimprove the estimates of change over time from theNHES.
Design of the Followup Survey
In the NHES:91, parents of 3- to 8-year-oldchildren were surveyed about the care andeducation of their children for the Early ChildhoodEducation (ECE) component. All eligible childrenwere sampled within each of the households forwhich a screening interview was completed. Whilethe child is the unit of analysis, the parent orguardian who was most knowledgeable about thechild's care and education was the respondent tothe ECE interview. In all, 13,894 interviews werecompleted in the ECE component. These
completed interviews represented a net responserate of 77 percent, the product of the screeningresponse rate of 81 percent and the extendedinterview (ECE) response rate of 95 percent.Brick et al. (1992b) presents additionalinformation on the NHES:91 ECE componentdesign, survey procedures, and unit and itemresponse.
As noted earlier, conducting a longitudinalfollowup survey with the respondents to the NHEShas some analytic advantages, but the feasibility ofsuccessfully accomplishing this data collection hadnot been evaluated prior to this experiment. Theexperiment involved drawing a random subsampleof children who were the subjects of completedNHES:91 ECE interviews, asking the parentrespondents a series of additional questions thatwould assist in locating them at a later date,attempting to locate the same respondents about 1year later, and conducting a brief followupinterview with the respondents to determine howwilling they would be to complete anotherinterview.
The subsample for the followup survey wasselected during the original interview in 1991.When the household was sampled in 1991 for theECE component, a subsample of children wasselected to participate in the followup study; 513NHES:91 interviews were completed for childrenselected for the followup study. A sample of about500 is sufficient to estimate a response rate to thefollowup with a standard error of less than 2percent. All 13,894 children whose parentscompleted initial interviews were eligible for thefollowup study, with the exception of oneconstraint; the subsample was restricted so that nomore than one child per household could besampled for the followup. Because telephoneinterviews can be time-consuming, particularly forbusy parents, limiting followup to a single child ineach household was used to reduce the timecommitment for parents and to contribute to a highfollowup response rate.
Since the NHES:91 sample was selected usingRDD methods, the only identifying information forthe sampled persons available at the time of the
4
initial interviews was their telephone number.Additional data items were asked in the NHES:91to facilitate locating the sampled persons at a latertime. These items included the respondent's fullname and address and the name, address, telephonenumber, and relationship of a contact personoutside the respondent's home who could beexpected to know how to locate the person in thefuture. Nearly 90 percent of the respondentsprovided their own full names and addresses.Booth and Johnson (1985) report similar levels ofresponse to items asking respondent information;89 percent of their respondents provided theirnames and addresses. However, only about two-thirds of the NHES respondents were willing togive the requested data about a contact person(Booth and Johnson report 72 percent cooperationin naming contact persons). A series of surveyprocedures was undertaken to locate and interviewthe sampled parents 1 year later. These proceduresare described below.
Survey Procedures for the Followup
The followup survey began in late April 1992 andended a month later, about 1 year after the initialNHES:91 data collection period. The surveyprocedures for a typical RDD survey were notappropriate for the followup survey becausespecific respondents to the NHES:91 were sought.Because those respondents could have changedtelephone numbers and addresses during the year,procedures were developed to attempt to locate andthen interview the NHES:91 respondents for thefollowup survey. In the followup interview, eachrespondent was asked to answer a limited subset ofitems about the sampled child.
Figure 1 presents an overview of the locating andinterviewing steps for the followup survey. Everysampled respondent was classified as having beenlocated or not, depending on whether or not it waspossible to verify that the respondent lived in ahousehold that was called during the followupsurvey. For all the respondents who were located,an additional status code was recorded indicatingwhether or not an interview was completed. Thetwo classifications (located and interviewed) are
key for evaluating the potential of conductingmeaningful followup surveys for future NHESadministrations.
Locating Procedures
The entire subsample of 513 persons was includedin the followup survey, regardless of whether thecontact information was available from theNHES:91 interview. That is, those persons whoprovided additional followup information wereincluded, as were persons who did not provide thisinformation. For all 513 respondents to theNHES:91, at least the first name of the parentrespondent was known. Most of the respondents(90 percent) gave their last name and address whenthese questions were asked in the supplemental setof items included in the NHES:91. Because thefull name and address were needed for somelocating procedures, the results of the locatingactivities are sometimes given separately for thosewho provided contact information and for thosewho did not.
Prior to making any telephone calls to therespondents, postcards were mailed to all thosewho provided their name and address in theNHES:91 interview. The purposes of the postcardwere to remind the respondents that they would beinterviewed again shortly and to obtain addressupdates from the Postal Service for any personswho might have moved during the last year.Exhibit 1 is a copy of the postcard. Theinformation from the postcards was used onlywhen the sampled person no longer lived at thesame telephone number and had to be located.
The first attempt to contact all 513 persons for theinterview was to call the telephone number usedfor the NHES:91 interview. Given that only 1 yearhad passed since the NHES:91 interview, arelatively large proportion of the respondents wasexpected to be located in a household with thesame telephone number. Depending on theoutcome of that call and the type of availableinformation on the respondent, one of severalpossible locating efforts was followed. Telephone
5
Figure 1.—Process of locating and interviewing respondents in the NHES:91 followup survey
numbers of followup respondents were identifiedwith the aid of five different sources, utilized asnecessary in the following sequence: the NHES:91telephone numbers, recorded messages with newtelephone numbers if the 1991 number was nolonger in service at followup, directory assistance,the contact people provided in the NHES:91interview, and a credit reporting service.
New technologies for locating persons using theInternet or CD-ROM files of phone listings maynow be available to enhance locating efforts, butthese methods were not in use at the time of thisstudy. While these new technologies do offer agreat advance, there are limitations associated withhow current the data are and the percentage ofpeople covered by the services. Continued
improvements in technology promise even moreeffective and accessible services in the future.
While many of the steps for locating respondentsare self-explanatory, others require more detaileddiscussion. One of these is the use of directoryassistance. In general, directory assistancetelephone calls were made only for thoserespondents who gave their full name and addresswhen interviewed in the NHES:91. The addressused for the directory assistance calls was eitherthe response from the NHES:91 interview or anyupdate of that address obtained from the postcardmailing. Directory assistance was also attemptedin a few cases where no address data existed,provided the last name of the respondent was
6
uncommon enough that there was some chance oflocating the appropriate respondent.
Another locating step that requires furtherclarification is the use of the contact person fromthe NHES:91 interview. When a call to directoryassistance did not yield a new telephone numberfor the respondent, the contact person identified inthe NHES:91 supplemental items was called toobtain a new number for the respondent. This wasdone at the same time as the directory assistancetelephone call.
The last source for locating respondents was acredit reporting service. The full name and addressof the respondent was required by the credit serviceto do this search, so those who did not give thesedata were excluded from this step.
Schedule and Interviewing
Interviewers who worked on the NHES:91 wereassigned to conduct the followup interviews. Since
they were already trained on the NHESprocedures, this enabled us to reduce the amount oftraining needed for the followup. Six experiencedinterviewers received additional training for thiswork.
A 4-hour training program included the followingactivities:
A review of the NHES:91 ECE component andthe followup feasibility test;
Procedures for conducting the followup;
Exercises showing each type of followup andthe sequence of activities; and
An interactive lecture to reacquaint theinterviewers with the questions.
The followup survey was done at Westat’sTelephone Research Center in Frederick,Maryland. The schedule of activities was asfollows:
Exhibit 1.—Postcard mailed to NHES:91 followup sample
NATIONAL HOUSEHOLD EDUCATION SURVEY
About 1 year ago, you or a member of your household participated in atelephone interview for a Department of Education study on the care andeducation of young children. During that interview, we told you that wewanted to contact you again in about a year to ask a few followup questionsabout your child. We will be making these telephone calls in the next severalweeks. As in the original interview, your participation is voluntary andconfidential. This brief followup interview will only take about 5 minutes ofyour time.
If you have any questions, please contact Kathryn Chandler at the U.S.Department of Education, National Center for Education Statistics, at 202-219-1767. You may also contact Mary Collins of Westat, Inc., who is managing thedata collection effort, toll-free at 1-800-937-8281.
Thank you for your continued assistance.
OMB 1850-0640
7
Date ActivityApril 8, 1992 Postcards mailed to sampleApril 22, 1992 Training of interviewersApril 22—May 31, 1992 InterviewingMay 8—May 31, 1992 Tracing through credit
service
Some activities were done continuously during thedata collection time period. These includedupdating addresses from postcard returns andmaking directory assistance calls as needed.
When a telephone number was identified for arespondent, the number was dialed at least seventimes over a 10-day period to attempt to locate thesampled person. If no contact with a householdmember was made during these repeated calls, anattempt was made to identify a new telephonenumber for the NHES:91 respondent untilavailable numbers from all sources wereexhausted. If the NHES:91 respondent was notfound, then the case was classified as not located.
If contact was made with a member of a householdand he or she verified that the NHES:91respondent or child was at this number, then thesampled child was classified as located. Multipleattempts were then made to interview the NHES:91respondent (if the NHES:91 respondent was nolonger a household member, but the child was,another responsible parent or guardian wasinterviewed). If after at least seven attempts aninterview was not completed, the case wasclassified as located but not interviewed.
Findings and Analysis
In this section, we report on the findings from thefollowup efforts. The findings describe the overallsuccess in locating and interviewing the NHES:91respondents. The results of specific efforts tolocate the persons also are presented. Theresponse experience for this feasibility study isplaced in context with a review of longitudinalresponse rates from two other national household-
based surveys. Each of these is an in-personsurvey, unlike the NHES, which uses a random-digit-dialing approach. Later, we discuss theimplications of these results for conducting alongitudinal survey of children with the RDDmethodology.
Locating and Response Rates
There were 513 persons eligible for the followupsurvey -- the total number of persons whocompleted the ECE interview in the NHES:91 andwere sampled for the followup. As a result of allthe efforts combined, 88 percent of the sampledNHES:91 cases were located and 80 percentcompleted followup interviews. Of the 452NHES:91 parent/guardian respondents who werelocated in the followup, 90 percent weresuccessfully interviewed (Table 1). These ratesare similar to those reported by Booth and Johnson(1985), who located 87 percent of their base yearrespondents, interviewed 90 percent of those whowere located, and interviewed 78 percent of theirtotal panel. Burgess (1989) reports that at least 80percent of survey respondents can be contacted formost surveys if tracing is used, and asserts thatsuccess rates in excess of 90 percent should beexpected; the survey method associated with theserates (telephone or in person) is not specified.
Table 1 shows the number and percent ofrespondents by response status, and for those whowere located, the number and percent for whominterviews were completed. About half of thecases where the respondent was located but nointerview was conducted were refusals, and almostall of the others were cases for which themaximum allowed number of attempts wasreached without completing the interview.
Response Status by Locating Characteristics
Table 1 also shows the distribution of responsestatus by whether or not the sampled personsprovided both full name and address information inthe NHES:91 interview. Nearly 90 percent (460 of
8
Table 1.—Number and percent of sampled persons, by response status and address information status
Response status Total With addressinformation1
Without addressinformation1
Number Percent Number Percent Number Percent
Total 513 100.0 460 100.0 53 100.0
Located 452 88.1 408 88.7 44 83.0
Completed interviews 409 90.5 374 91.7 35 79.5
Not completed interview 432 9.5 34 8.3 9 20.5
Not located 61 11.9 52 11.3 9 17.0
1Having address information means the respondent provided both full name and address in the NHES:91 interview.2Of the 43 interviews that were not completed, 21 were refusals, 19 were not reached after multiple attempts, and 3 were notcompleted for other reasons.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.
513) of the sampled persons provided thisinformation. Both the ability to locate the sampledrespondents and their willingness to complete theinterview once located appear to be greater forthose who gave full name and address informationin the NHES:91 interview, but the sample size istoo small to conclude this from this study.
Another key locating characteristic is whether thesampled person was located at the same telephonenumber used in the NHES:91 interview. Table 2shows the response distribution of the sampledpersons by whether or not they were located at thesame telephone number in 1991 and 1992.Slightly more than three-fourths (77 percent) of the
Table 2.—Number and percent of sampled persons, by response status and telephone location
Response status Total At same telephone number Not at same telephonenumber
Number Percent Number Percent Number Percent
Total 513 100.0 394 100.0 119 100.0
Located 452 88.1 394 100.0 58 48.7
Completed interview 409 90.5 357 90.6 52 89.7
Not completed interview 43 9.5 37 9.4 6 10.3
Not located 61 11.9 - - 61 51.3
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.
9
sampled persons were located by dialing the sametelephone number. All of the persons who werenever located are categorized as not being at thesame telephone number based on the assumptionthat they would have been located if they still hadthe same number. Although some of them actuallymay have been at the same telephone number, it islikely that only a very few persons were in thatcategory because repeated calls were made to theoriginal number if no other telephone numberswere identified for the respondent. Of the sampledpersons who were at the same telephone number,91 percent completed the followup interview. Only44 percent of those not at the same telephonenumber completed the interview (52 out of 119).However, 90 percent of those not at the sametelephone number who were located completed theinterview. Thus, the difference in completion ratesat this stage was the inability to locate the persons.
Table 3 shows the distribution of response statusby both the persons' willingness to provide nameand address in the initial interview and by whetheror not they were located at the same telephonenumber. The numbers are relatively small forthese tabulations and show little of substance.
Response Status by Tracing Efforts
Locating the sampled respondent was the first andmost important step in completing the followupinterview. As shown in Table 3, 119 of the 513sampled persons were not at the same telephonenumber and required some type of locatingactivity. For 113 of these persons, some type oflocating effort could be attempted, while for 6persons nothing could be done because of the lackof information on even the name of the sampledperson. Because of their importance, more detailedresults of applying the locating procedures arediscussed below.
If the person was not located at the originalnumber, the first possible step to locate arespondent involved recorded messages thatprovided new telephone numbers. Only 18telephone numbers had recordings with newtelephone numbers: 13 of these respondents werelocated, 11 produced completed interviews, and 5were not located (not in table).
Table 3.—Number of sampled persons, by response status, telephone location, and addressinformation
Response status At same telephone number Not at same telephone number
With addressinformation*
Without addressinformation*
With addressinformation*
Without addressinformation*
Total 359 35 101 18
Located 359 35 49 9
Completed interview 322 35 43 9
Not completedinterview
37 0 6 0
Not located 0 0 52 9
* Having address information means the respondent provided both full name and address in the NHES:91 interview.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.
10
Before the results from the other locating activitiesare presented, some factors associated with theresults must be considered. First, although thelocating activities were done sequentially, theactivity associated with ultimately locating therespondent cannot specifically be identified. Forexample, the Postal Service may have returned apostcard with a new address for a respondent afterthe credit searches for the person had begun. Inthis case, directory assistance for the person wouldhave been attempted immediately using the newaddress. In another example, a source may haveidentified a telephone number that had beenattempted previously. The number might havebeen temporarily disconnected or the respondentmight have been away during the earlier callingperiod; when the number was dialed again, therespondent was located. It is unclear how toassociate a source with locating the respondent inthese and many other situations.
Because of the difficulty of linking the source ofthe telephone number to locating and interviewingthe respondent, it is not possible to fully assess theeffectiveness of each source of locatinginformation. Instead, the final results of locatingand interviewing are presented for all the
respondents who were subjected to the varioustypes of locating activities. The discussion beginswith the postcards returned by the Postal Servicefor respondents who were no longer at their 1991address and ends with the credit bureau searchefforts.
All 460 sampled persons who provided a name andaddress in the NHES:91 were mailed a postcard(Exhibit 1). Postcards were returned by the PostalService for 59 of these persons (13 percent). Ofthe postcards that were returned, 44 contained newaddress information. Other people weresuccessfully interviewed, although 61 percent werelocated. After dialing the original telephonenumber and using data from the returned postcard,directory assistance was dialed for those sampledpersons who were still not located but hadprovided their name and address in the NHES:91interview. As shown in Table 4, directoryassistance was used for 97 cases (21 percent).Only 44 percent of those persons were located and37 percent completed the followup interview.
The results of directory assistance shown in Table4 also include telephone calls made to contactpersons whom respondents identified in the
Table 4.—Number and percent of sampled persons, by response status and tracing activity
Tracing activity Response status
Total Located Located,completedinterview
Located, notcompletedinterview
Not located
Number Number Percent Number Percent Number Percent Number Percent
Postcard returns 44 27 61.4 23 52.3 4 9.1 17 38.6
Directory assistance/contact person 97 43 44.3 36 37.1 7 7.2 54 55.7
Credit searches
with new locating data 15 4 26.7 3 20.0 1 6.7 11 73.3
without new locating data 33 6 18.2 5 15.2 1 3.0 27 81.8
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.
11
NHES:91 interview. The results of these activitiescannot be separated, because the telephoneinterviewers immediately attempted to reach thecontact persons if no new telephone informationwas obtained from directory assistance for therespondent. Contact persons were named for two-thirds of the 513 sampled persons, but calls wereonly needed for 44 of these persons. Twenty-eightpostcards returned by the Postal Service had notessuch as "Address unknown." Table 4 shows thedistribution of the response status for the 44persons for whom the postcard mailing resulted inobtaining a new address. Only 52 percent of theseof these calls resulted in a new telephone numberto contact the sampled respondent. Of theremaining 16 contact persons called, 12 said thatthey would call back on the Westat toll-freenumber after they spoke to the sampled respondentbut ultimately did not call back, and the other 4refused to provide any new data.
The final locating strategy was the use of a creditbureau search for the sampled persons who had notbeen located by any of the previous efforts. Forty-eight names were sent to the credit service, and 15were returned with different information thanprevious sources had provided. In many of the 33other cases, the credit searches resulted in addressinformation that had already been obtained fromone of our previous tracing efforts. Only 27percent of the sampled persons for whom newlocating information was obtained from the creditsearches were located, and 20 percent completedthe interview.
After all the locating activities were completed,final attempts were made to contact all unresolvedcases. In most cases, these efforts involved dialingthe NHES:91 telephone number again. If thepersons had been located but not interviewed, thetelephone number at which they were located wasdialed again. These efforts resulted in somecompleted interviews. This is also the reason thatfive completed interviews are shown in Table 4 forthe cases in which the credit searches did not resultin new locating data. Some persons located andinterviewed in this process had telephones that hadbeen temporarily disconnected, and others had
been away from home during the initial fieldperiod.
A summary of all the locating efforts attempted ispresented as Table 5. This table gives the numberand percent of persons by whether or not they werelocated and interviewed for each of the differenttypes of locating activities attempted. The locatingefforts were essential in improving the responserates to the survey. As a result of the locatingactivities, about 10 percent (53 of 513) of therespondents were located and 8 percent (43 of 513)were interviewed. The postcard returns and thedirectory assistance calls (including using thecontact persons from the NHES:91 interview) wereresponsible for finding most of these respondents.
These locating activities are relatively low costefforts and should be included in any followupsurveys.
The credit bureau searches are more expensive anddid not result in locating many more of therespondents. The utility of the credit bureausearches needs to be closely considered in futurestudies. The cost of conducting the searches mustbe weighed against a possible 1 to 3 percentincrease in the response rate. The direct cost ofcredit bureau searches to locate a person'stelephone number varies depending on the natureof the search. Typically, it costs about $3 to $10per searched name, with other costs associatedwith handling and recording the data retrieved fromthe service. For followup surveys where the lossdue to nonresponse is a major problem, it may beuseful to incur the credit bureau search costs evenfor a relatively small increase in response rates.As noted earlier, new technologies such as theInternet may provide other methods for locatingpersons at reasonable cost, but these were not inuse in 1992 when this survey was conducted.
Implications for Longitudinal Analysis
The findings discussed above provide a frameworkfor evaluating the feasibility of conducting afollowup telephone survey with parents of youngchildren when the sample is selected using RDD
12
Table 5.—Number and percent of sampled persons, by response status and all types of locating activities
Response status Type of locating activity Number ofpersons
Percent
Postcard returned Directoryassistance
Credit searches
Located none none none 399 88.3none called none 20 4.4none called new data 3 0.7none called no new data 4 0.9return none none 8 1.8return called none 15 3.3return called new data 1 0.2return called no new data 2 0.4
All located - - - 452 100.0Located and completed none none none 366 89.5
none called none 16 3.9none called new data 2 0.5none called no new data 3 0.7return none none 5 1.2return called none 14 3.4return called new data 1 0.2return called no new data 2 0.5
All located and completed - - - 409 100.0
Located and not completed none none none 33 76.7none called none 4 9.3none called new data 1 2.3none called no new data 1 2.3return none none 3 7.0return called none 1 2.3
All located and notcompleted
- - - 43 100.0
Not located none none none 1 1.6none none no new data 1 1.6none called none 18 29.5none called new data 7 11.5none called no new data 17 27.9return none none 2 3.3return called none 2 3.3return called new data 4 6.6return called no new data 9 14.8
All not located - - - 61 100.0
NOTE: Percentages may not add to 100 because of rounding.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.methods. The two major issues that must beaddressed in using the NHES for a longitudinal
followup survey are coverage bias and unitnonresponse bias.
13
Coverage bias arises because not all eligiblepersons live in households with telephones.Previous research on telephone coverage for youngchildren indicated that the coverage bias in theestimates for this population will be relativelysmall if appropriate weighting adjustments aremade during estimation (Brick et al. 1992a). Ofcourse, estimates for domains that contain adisproportionately high fraction of theundercovered population may be subject to moresevere biases. Unit nonresponse bias is the moreimmediate concern of this research. The size of thebias due to nonresponse is related to the percentageof the sample that did not respond to the surveyand the differences in the characteristics of therespondents and the nonrespondents. One way ofexpressing this nonresponse bias as a fraction ofthe response rate and the difference incharacteristics is:
where yr is the estimate based on responses onlyfrom respondents, r is the number of respondents,n is the total sample size, Yr the value of thecharacteristics for respondents, and Ynr is the valueof the characteristics for nonrespondents. This
expression shows that as the response rate ( )rn
decreases and the difference between respondentand nonrespondent characteristics increases, thebias of the estimate can become large. Kalton,Kasprzyk, and McMillen (1989) discuss thepotential impact of nonresponse bias forlongitudinal surveys. They point out thedeleterious effect on the estimates as response ratesdecrease and differences between respondents andnonrespondents increase.
The role of nonresponse in the bias of the estimatesis the reason this experiment was developed toassess the potential rate in a longitudinal study.Differences in characteristics of respondents andnonrespondents are harder to quantify because thedifference depends on both the specific itemsconsidered and the reasons for the nonresponse.
For example, some characteristics, such aspoverty, low maternal education, and high familymobility are educational risk factors for children(Zill et al. 1995; Sattler 1992). To the extentfactors such as these are associated with surveyresponse and the ability to locate the subject duringlongitudinal followup, they may be sources of bias.
Since no data were collected from nonrespondentsin the followup, it is not possible to compare theircharacteristics with those of the respondents basedon the followup interview. However, the initialNHES:91 interview was completed by all sampledcases, and these characteristics can be examined toprovide some insight into the correlates ofnonresponse in the followup. Of course, importantcharacteristics such as moving to a different homeand changing family composition are not includedwhen using only the data from the initial interview.
Data from the NHES:91 interviews were used tocompute estimates of the differences between thosewho did and did not respond to the followup. Theestimates were derived using the weights from theNHES:91 to account for the probability ofselecting the persons for that survey. Theseestimates differ slightly from unweighted analysis.For example, the estimated response rate for thefollowup using the weights is 76 percent while theunweighted response rate is 80 percent.
Table 6 presents estimates of the response rates tothe followup by the demographic and earlychildhood characteristics from the NHES:91interview. Estimated differences between responserates to the followup are subject to very largesampling errors because the sample sizes aresmall. Thus, even moderately sized differences inresponse rates cannot be distinguished due to thelarge sampling errors. A few of the differencesbear discussion because they are so large that they
Bias yrn
Y Yr r nr( ) ( ( )= − −1 (1)
14
Table 6.—Followup response rates by child and household characteristics
Characteristic Unweighted number of cases1 Weighted response rate
Total
Race White Black Other
Hispanic origin Hispanic Non-Hispanic
Home Type Single family or townhouse Apartment Other type
Home ownership Own Rent Some other arrangement
Income $20,000 or less $20,001 to $40,000 More than $40,000
Parents' highest education2
High school or less Some college/vocational/technical Bachelor's degree or more
Mother's first language English Spanish Other
Family receives daily newspaper Yes No
Census region of residence Northeast Midwest South West
Read to child in past week Yes No
Child weighed 5.5 pounds or more at birth Yes No
512
3957245
103409
3987836
34215218
118206188
193160156
4376213
301210
9512819396
46347
47735
76
854980
7279
845674
8666--3
687888
727985
7875--3
8175
85777578
7880
7880
1One case is excluded from this analysis due to missing values.2Highest education of the child's parent(s) or guardian(s) living in the household.3Estimates based on fewer than 30 cases (unweighted) are not shown and are indicated by --.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey(NHES), followup survey, 1992.
15
are statistically significant despite the largesampling errors.1
The most important difference in followupresponse rates was between white and blackpersons. The estimated response rate was 49percent for blacks and 85 percent for whites. Withsuch a large difference between the response ratesof blacks and whites, even small differences in thecharacteristics of the respondents could result inlarge biases. Thus, the nonresponse bias forestimates by race could be very substantial.
It is interesting to note that the estimates byHispanic origin were not large or statisticallysignificant. The estimated response rate forHispanics was 72 percent as compared to 79percent for non-Hispanics.
Another characteristic that was statisticallysignificant was whether the person lived in anapartment or single family house. Those who livedin an apartment had an estimated followupresponse rate of 56 percent, while those who livedin a single family house had an estimated responserate of 84 percent. Similarly, persons who ownedtheir homes had higher response rates than renters(86 percent versus 66 percent). This result isconsistent with the hypothesis that those who aremost likely to move, such as renters and apartmentdwellers, are underestimated in longitudinalsurveys due to nonresponse bias. Allen andPetroni (1994) report on research to account formovers explicitly using different nonresponseadjustment methods, although they were notable toidentify any adjustments that reduced attrition biasin their study.
Persons in households with incomes over $40,000annually (88 percent) and those in households withincomes of $20,000 or less (68 percent) aresignificantly different. Similarly, children whoseparents' highest education is a bachelor's degree ormore have higher response rates than those whose
1 The sampling errors for the differences in response rates
were calculated by multiplying the simple random
sampling error ( / )pq n by the square root of the mean
design effect for the NHES:91 ECE component.
parents' highest education is high school or less (85percent versus 72 percent). Because education andincome are associated with home ownership and,hence, with mobility, these differences are notsurprising.
Most of the selected variables of substantiveimportance for early childhood education (e.g.,how often the child was read to by a parent,whether the child weighed more than 5.5 pounds atbirth, or whether the child was read to in the pastweek) had relatively small differences in estimatedresponse rates. Of course, if the sample size werelarger, many of these smaller differences would bestatistically significant if the observed differencesremained.
Longitudinal Response Rates
The response rate for a longitudinal survey is thecompletion rate for the initial survey multiplied bythe followup completion rate, assuming that onlythose who responded to the initial survey arefollowed. In the NHES application, the followupresponse rate is the product of the response rate forthe NHES:91 ECE interview and the completionrate for the followup survey. The response rate forthe NHES:91 ECE interview was 77 percent (theresponse rate to the ECE was the product of the 81percent screening completion rate and the 95percent completion rate for the ECE interview).Thus, the overall response rate for longitudinalanalysis using the followup data is 62 percent (77percent for the original survey times 80 percent forthe followup survey).
It might be possible to increase the followupresponse rate by increasing the percentage ofrespondents who are located. Often, the greaterthe time between contacts, the greater thepercentage of cases that are lost to the study.Burgess (1989) reviews some approaches totracing respondents in longitudinal surveys. Onestrategy to potentially increase the locating ratewould be to have semiannual or quarterly postcardmailings. This relatively low-cost approach wouldincrease the likelihood of obtaining change-of-address information from the Postal Service in a
16
timely way and would also remind the respondentof the study. By contrast, interim telephonecontacts between data collection phases would beconsiderably more costly, but may be no moreeffective.
It is interesting to speculate on what the responserate might be if the followup completion rates weresomewhat different. For example, if all thepersons who were located in the followup survey(88 percent) were successfully interviewed, thefollowup response rate would be 68 percent (77percent times 88 percent). This is only 6 percentgreater than the observed rate. If the locating ratewere raised from 88 percent to, say, 95 percent,and the same 90 percent response from locatedpersons was obtained, the followup response ratewould be 65 percent (77 percent times 86 percent),an increase of only 4 percent. Future followupefforts should seek to maximize both the locatingrate and the response rate, but the potential forincreasing the overall response rate using thesemethods is limited.
Clearly, the followup response rate is depressed bythe 81 percent completion rate for the screening ofhouseholds in the initial NHES:91. Severalattempts have been made to increase this rate in theNHES, for example, by refielding cases that havehad the maximum number of attempts in the studyprotocol, and research in this area continues. Forexample, in the NHES:96, an advance mailing wassent to all households for which an address couldbe obtained from a commercial vendor. However,attaining completion rates of 85 percent or higherin this type of general population survey is not yetfeasible. Assuming the screening rate is 85 percentand all the other rates are as observed, thelongitudinal response rate would be 65 percent (85percent times 95 percent times 80 percent). Onceagain, this is higher than the observed rate, butonly marginally. Conversely, if design decisionsfor the study lead to lower screener response rates,the net response rate for a longitudinal study wouldbe decreased.
In essence, this analysis suggests that thelongitudinal response rate for a RDD survey likethe NHES would probably be between 62 and 65
percent. A 70 percent rate appears to be the upperlimit, unless better techniques for gainingparticipation are discovered and implemented. Tounderstand how these response rates compare withthe rates that might be attained using differentmodes of sampling and data collection, theresponse rates for longitudinal household andcohort surveys conducted by the Census Bureauand the Ohio State University are described below.Although these comparisons are useful, it isimportant to recognize that the demands placed onthe respondents in these surveys vary significantlyand may affect some of the results.
SIPP Response Rates
The Survey of Income and Program Participation(SIPP) is a longitudinal household surveyconducted by the Bureau of the Census. Since it isan important household-based panel survey, SIPPprovides an appropriate comparison to the NHESexperiment on the feasibility of collectinglongitudinal data.
Several SIPP surveys have been conducted sincedata collection started in 1984. The surveyconsists of an initial interview and followupsurveys conducted at 4-month intervals. Thesedata collections are called waves. For example, ifthe Wave 1 interview takes place in January, thenthe Wave 2 interview should be conducted in Mayof the same year. More details on the design ofSIPP and the data we used on response rates canbe found in Jabine et al. (1990).
The first wave of data collection in SIPP is alwaysdone by personal visit to the household.Subsequent data collection is done primarily byphone but may also be done in person. Followupdata are not collected for households that do notparticipate in the Wave 1 data collection. Forcomparative purposes, we will use the ratesreported for the 1988 Panel of SIPP, i.e., thesurvey that began data collection in February1988. The overall response rates for the eightwaves of this survey are given in Table 7. Itshould be recognized that the population of interestin SIPP is not restricted to households with youngchildren.
17
Table 7.—Response rates from the 1988 Panel of the Survey of Income and Program Participation
Response Condition Response rate
Responded to all 8 interviews 67.0%
Responded to all interviews prior to death 1.5
Responded to all interviews except when: in an institution in the Armed Forces abroad
0.71.40.6
Other panel members 0.2
Total 71.4
SOURCE: Lepkowski, Kalton, and Kasprzyk (1989)
The response rates reported for this survey aremuch higher than those that could be expectedusing an RDD screening sample. The maindifference is the completion rate for the initialsurvey. The Wave 1 rate for SIPP was 93 percent,while the NHES:91 ECE rate was 77 percent.While the difference in the rates is largely theresult of the different modes of sampling and datacollection, the Census Bureau's response rates aregenerally higher than those achieved by other datacollection organizations.
With personal interviews there are moreopportunities to obtain participation, includingactivities such as speaking with neighbors, and it iseasier to demonstrate the importance of thesampled person's cooperation. The Bureau's earlyexperiments with RDD samples had response ratesthat are more consistent with the NHES responserates (Biemer, Chapman, and Alexander 1985;Fitti 1979).
The completion rates for followup surveys are alsogreater in SIPP than in the NHES experiment. Forexample, 94 percent of the Wave 1 respondentscompleted the Wave 2 interview in SIPP, whileonly 80 percent of the initial NHES:91 samplecompleted the followup interview. This is largely afunction of being able to locate the respondents. Inthe followup to the NHES:91, 90 percent of thoselocated were interviewed. The lack of information(including full name and address) to locate thesample persons in a RDD survey is a majorimpediment to improving the completion rates for
the followup. The fact that the SIPP followup wasconducted only 4 months after the initial interviewwhile the NHES followup was a year later alsoaffects this comparison. Since the NHES followupwas conducted 1 year after the initial interview, itis not possible to evaluate the impact of the timebetween interviews on the rates. Alternativeprocedures, such as interim postcard mailings andtelephone contacts discussed earlier could reducethe numbers of cases lost to the study, but also addcosts for a large-scale effort.
NLSY Response Rates
The National Longitudinal Survey of Youth(NLSY) is a longitudinal survey sponsored by theU.S. Bureau of Labor Statistics, U.S. Departmentof Labor that is conducted by the U.S. Bureau ofthe Census and the National Opinion ResearchCenter. The survey is directed by the Ohio StateUniversity. In the NLSY, interviews are conductedwith a sample of persons born January 1, 1957through December 31, 1964. The base year for thesurvey was 1979. Annual followup surveys areconducted with this cohort.
Table 8, below, gives the response rates for theNLSY civilian sample2 for the years 1979 through
2 The NLSY included a civilian sample, a supplemental
civilian sample, and a military sample. Also, in 1986, achild sample was selected from among the children ofwomen in the civilian samples. For ease of presentation,the civilian sample figures are presented here. Retentionrates were similar for the supplemental civilian sample.
18
Table 8.—Response and sample retention rates for the civilian sample of the National Longitudinal Survey ofYouth: 1979 to 1991
Survey year Interviewscompleted
Percent of sampleretained
Response rate
1979 (Base year) 6111 -- 81.8
1980 5873 96.1 78.6
1981 5892 96.4 78.8
1982 5876 96.2 78.6
1983 5902 96.6 79.0
1984 5814 95.1 77.7
1985 5751 94.1 76.9
1986 5633 92.2 75.4
1987 5538 90.6 74.1
1988 5513 90.2 73.7
1989 5571 91.2 74.6
1990 5498 90.0 73.6
1991 5556 90.9 74.3
NOTE: The base year response rate is the product of the 91.2 percent screener completion rate and the 89.7 base yearinterview completion rate for the civilian sample. Retention rates reflect the percent of completed base year respondents whowere interviewed the subsequent years, excluding ineligibles. Yearly response rates reflect the sample retention ratemultiplied by the base year response rate.
SOURCE: Center for Human Resources Research, Ohio State University. NLS User's Guide: 1993. Tables 1.1.1 and 1.1.3.
19
1991. The retention rate for each followup phasemultiplied by the base year response rate gives theresponse rate. The initial interviews wereconducted in person, as were annual followupinterviews through 1986. At that time, a changewas made to a shorter interview administered bytelephone. By the time the shift to telephoneinterviewing began, eight annual cycles ofinterviews had been conducted in person. At eachfollowup point, efforts were made to interview allpersons who completed the base year interview,whether or not the person had responded to theprevious year's followup interview.
The NLSY started with a base year response rateof 82 percent in 1979. Over the 12 years from1979 through 1991, the response rate declinedslightly, and the 1991 response rate was 74percent.
The NLSY response rates are higher than theNHES rate because of the higher base yearresponse rate and the higher locating rate. Onenotable difference between the NHES and theNLSY is in the screening rate; the NLSY screenercompletion rate was 10 percent higher than theNHES:91 screening response rate. It is notuncommon for personal interview surveys to havehigher initial response rates (Drew, Choudry, andHunter 1988). The higher base year rate isessential to obtaining the higher response rates inthe followup surveys. In addition, the NLSY has ahigher completion rate at the followup stage thanthe NHES. Like the screening rate, this isprobably related to the fact that the base yearinterview was a personal interview, so that addressinformation was available for all cases at the baseyear data collection. In addition, in-personinterviews tend to establish greater rapport andmay have led to greater willingness to continue toparticipate in the study. The NLSY, unlike theSIPP and NHES, uses a small incentive ($10)upon completion of the interview.
Conclusions and Implications
The results of the efforts to locate and interviewthe respondents for the NHES:91 longitudinal
experiment indicate that there are significantquestions about the feasibility of conducting alongitudinal survey using an RDD sample for thebaseline. The main problem is nonresponse bias.The response rate for the base year of the ECEinterview for the NHES:91 was 77 percent, andcompounded with the locating and interviewingcompletion rates for the followup survey, theoverall response rate was about 62 percent. Biasesin surveys with rates this low are of great concern.
Booth and Johnson (1985), who started with alower base year response rate but obtained similarlocating and interviewing rates, conclude that"panel studies based on random-digit-dialingsamples can yield samples comparable to thosegathered by more traditional methods." Theauthors of this report take a more cautious view.Although some longitudinal surveys with responserates even lower than the NHES feasibility studyare conducted, alternative modes of data collectionshould be fully studied before using the NHES forthis type of data collection.
Although the nonresponse rate is due to both theinitial nonresponse and the nonresponse at thefollowup, bias for the followup may be a moreserious concern. The nonrespondents for the
followup were largely persons who have moved (orat least changed telephone numbers) in the lastyear. This group of more mobile households maybe very different from the responding households,which are less mobile. This could result in largerbiases, especially because residential mobility is arisk factor for young children. Furthermore, theresponse rates for the followup were different forsome important groups. In particular, the responserate for blacks was very low and could result inlarge biases by race.
Some changes in the technology of telephonesurveys may influence decisions about futuresurveys. Methods for locating persons based onlyon their telephone numbers may become availableand would help to increase the rate at whichpersons can be located, although this change alonemay not be sufficient for making the longitudinalapproach viable in the NHES. If the initial
20
screening response rates can also be significantlyimproved, then followup surveys using the NHESas the first wave should be reconsidered.
Cost Considerations
When evaluating the desirability of implementing alongitudinal study, cost factors must also beconsidered. The feasibility study was not designedto provide cost estimates for a full-scale study. Avery brief and simple interview was used, whichrequired a limited investment in the development ofthe CATI system and used only experiencedinterviewers. The costs associated with a longerand more complex instrument would be greater,particularly if skip patterns or edits based onresponses from previous interviews were included.While this followup study used a fairly standardprotocol for data collection, full-scale NHEScollections typically include more elaborate refusalconversion methods and the refielding of cases thathave reached the maximum number of attemptscalled for in the study protocol.
In comparison with a base year random-digit-dialsurvey, a followup survey does not incur thesubstantial costs associated with screening largenumbers of telephone numbers to identifyhousehold numbers and eligible householdmembers. A longitudinal study would, however,incur substantial costs associated with conductingtracing activities to maximize the percentage of thesample that is located. Some of these activities, asnoted previously, are relatively inexpensive. Forexample, mailing a postcard with "addresscorrection requested" was found to be useful andinexpensive. Making directory assistance callswhen the original telephone number is no longervalid is also a low cost method, as is making callsto contact persons named by the respondent in theoriginal interview. More costly tracing efforts,such as credit bureau checks, can be productive.However, it is most cost effective to schedule thefollowup field period such that other, less costlymethods are exhausted before the cost of creditbureau checks is incurred. As mentioned, newtechnologies for locating persons may be useful inthis regard.
Burgess (1989) suggests that several rounds oftracing efforts conducted periodically betweensurvey cycles may ultimately be more cost-effective than extensive (and perhaps lesssuccessful) tracing efforts at the followup surveypoint. While Burgess' example is for a multi-yeartime frame, this may also be germane to a 1-yearfollowup as in the NHES feasibility study. Thelocating rate might be increased by relatively low-cost interim tracing activities, such as mailing apostcard with "address correction requested" abouthalf-way between the initial survey and thefollowup point.
The potential cost of longitudinal studies developedfrom the NHES could vary widely. These costsare dependent on the specific design elements ofthe study and the strategies required to implementthose design elements. While it is very likely thatthe costs would be much smaller than surveys thatare conducted in person, it is difficult to quantifythe cost savings associated with this approach.
Alternative Assessment of Change Over Time
This research into a longitudinal analysis does notimply that the NHES cannot be used to preciselymeasure change over time in the population.Rather, it addresses longitudinal approaches thatrequire contacting the same sampled personsrepeatedly. Other ways of measuring change in thepopulation are feasible. For example, changes inthe characteristics of the population can bemeasured with independent cross-sectionalsurveys. With such a sample design, the varianceof the estimated measure of change is the sum ofthe variances for the estimates at the two points intime. The variances of estimates of change arethus almost always larger than would be the case ifthe same persons were followed and interviewed.
Alternative methods for reducing the variance ofestimates of change can be employed without usingthe same sample of persons over time. Theprecision of estimates of change can be improved ifthe samples from the two points overlap. Since theoverlapping sampled units tend to have largepositive correlations for most characteristics of
21
interest, the precision of estimates of change can beimproved by this design.
One way of inducing this correlation is to reuse thesampled clusters from the first stage of the RDDsample. These are the banks of 100 consecutivetelephone numbers from which additionaltelephone numbers are generated using thestandard or modified Waksberg methods of RDDsampling (see Brick and Waksberg (1991) formore details on these RDD designs). An openquestion is the size of the correlations that wouldbe obtained using this method. If the correlationswere relatively small, then the precision of theestimates would not be significantly improved.While the size of these correlations currently isunknown, it is likely to be small for the NHESpopulation of young children because this group isrelatively rare within a cluster. In addition, thismethod is not consistent with the move to list-assisted methods of RDD sampling (Casady andLepkowski 1993; Brick et al. 1995) in the NHES,although the list-assisted method could be modifiedto achieve many of the same benefits.
Another approach that might provide highercorrelations and improved precision for estimatesof change is to reuse the sampled telephonenumbers. In this design, sampled respondents arenot followed from year to year, but many of thepeople (about three-fourths based on the followupresults) will be at the same telephone number. Thenumber of persons who are sampled in bothsurveys will be less than this because of changes inhousehold composition.
This approach has two sample components: asample of the telephone numbers dialed in theoriginal survey, and a new sample of telephone
numbers. The new sample is needed to eliminatethe bias associated with persons currently inhouseholds that were not valid telephone numberswhen the original sample was selected. Thenumber of households sampled from the twocomponents is allocated based on the desiredprecision of the estimates. If estimates of changeare more important than the cross-sectionalestimates, more of the sample will be allocated tothe numbers dialed in the original survey. Someadditional information would also have to beobtained from the households to make nationalestimates.
This approach improves the precision for estimatesof change, but it is not a pure longitudinal sampledesign in which the sampled persons are repeatedlyinterviewed. It does avoid the major nonresponseproblem of being unable to locate the sampledpersons 1 year later. The other reasons fornonresponse will still affect respondents in thisdesign. In particular, concerns about respondentsbecoming reluctant to respond to the survey afterthey are repeatedly interviewed still need to beevaluated. Given the two NHES goals of havinghigh response rates and improving estimates ofchange, the reuse of the sampled telephonenumbers is an approach that could be testedempirically. A subset of the sampled telephonenumbers could be included in a future NHESsurvey, and the response rates for the newlysampled and repeatedly sampled numbers could becomputed. This experiment would also providedata on the size of the expected correlations andthe differences in the expected costs of datacollection for the two treatments. This method ofusing the NHES to monitor changes over timeappears more promising than retaining the samesampled persons over time.
22
References
Allen, T., and Petroni, R. (1994). Mover nonresponse adjustment research for the Survey of Income andProgram Participation. Proceedings of the American Statistical Association Section on SurveyResearch Methods. Arlington, VA: American Statistical Association, 662-667.
Bailar, B. (1989). Information needs, surveys, and measurement errors. In D. Kasprzyk, G. Duncan, G.Kalton, and M.P. Singh (Eds.), Panel Surveys, 1-24. New York: John Wiley and Sons.
Biemer, P., Chapman, D., and Alexander, C. (1985). Some research issues in random-digit-dialing andestimation. Proceedings of the First Annual Research Conference of the U.S. Census Bureau.Washington, DC: U.S. Bureau of the Census, 71-86.
Booth, A., and Johnson, D.R. (1985). Tracking respondents in a telephone interview panel selected byrandom digit dialing. Sociological Methods and Research, 14:53-64.
Brick, J.M., Burke, J., and West, J. (1992a). Telephone Undercoverage Bias of 14- to 21-year-olds and 3- to 5-year-olds. Technical report. NCES Publication No. 92-101. Washington, DC: U.S.Department of Education, National Center for Education Statistics.
Brick, J.M., Chandler, K., Collins, M.A., Celebuski, C.A., Ha, P.C., Hausken, E.G., Nolin, M.J., Owings,J., Squadere, T.A., and Wernimont, J. (1992b). National Household Education Survey of 1991:Preprimary and Primary Data Files User's Manual. NCES Publication No. 92-057.Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Brick, J.M., and Waksberg, J. (1991). Avoiding sequential sampling with random digit dialing. SurveyMethodology, 17(1), 27-41.
Brick, J.M., Waksberg, J., Kulp, D., and Starer, A. (1995). Bias in list-assisted telephone samples.Public Opinion Quarterly, 59(2), 196-217.
Burgess, R. (1989). Major issues and implications of tracing survey respondents. In D. Kasprzyk, G.Duncan, G. Kalton, and M.P. Singh (Eds.), Panel Surveys, 52–74. New York: John Wiley andSons.
Casady, R.J., and Lepkowski, J.M. (1993). Stratified survey designs. Survey Methodology,19(1), 103-113.
Center for Human Resources Research. (1993). NSL User's Guide: 1993. Columbus, OH: Center forHuman Resources Research, Ohio State University.
Drew, J.D., Choudry, G.H., and Hunter, L. (1988). Nonresponse issues in government telephone surveys.In R.M. Groves, D.P. Biemer, L.E. Lyberg, J.P. Massey, W.L. Nicholls, and J. Waksberg (Eds.),Telephone Survey Methodology, 233-246. New York: John Wiley and Sons.
Duncan, G., and Kalton, G. (1987). Issues of design and analysis of surveys. International StatisticalReview, 55(1), 97-117.
23
Fitti, J. (1979). Some results from the Telephone Health Interview System. Proceedings of the AmericanStatistical Association Section on Survey Research Methods. Arlington, VA: AmericanStatistical Association, 244-249.
Fitzgerald, R. (1994). Descriptive Summary of 1989-90 Beginning Postsecondary Students: TwoYears Later. NCES Publication No. 94-386. Washington, DC: U.S. Department of Education,National Center for Education Statistics.
Groves, R.M., and Lyberg, L.E. (1988). An overview of nonresponse issues in telephone surveys. InR.M. Groves, D. P. Biemer, L.E. Lyberg, J.P. Massey, W.L. Nicholls, and J. Waksberg (eds.),Telephone Survey Methodology, 191-211. New York: John Wiley and Sons.
Ingels, S.J., Dowd, K.L., Baldridge, J.D., Stipe, J.L., Bartot, V.H., Frankel, M.R., Owings, J., and Quinn,P. (1994). National Education Longitudinal Study of 1988 Second Followup: StudentComponent Data File User's Manual. NCES Publication No. 94-374. Washington, DC: U.S.Department of Education, National Center for Education Statistics.
Jabine, T., Petroni, R., and King, K. (1990). SIPP Quality Profile. Washington, DC: U.S. Departmentof Commerce, Bureau of the Census.
Kalton, G., Kasprzyk, D., and McMillen, D. (1989). Nonsampling errors in panel surveys. In D.Kasprzyk, G. Duncan, G. Kalton, and M.P. Singh (Eds.), Panel Surveys, 249-270. New York:John Wiley and Sons.
Lepkowski, J., Kalton, G., and Kasprzyk, D. (1989). Weighting adjustment for partial nonresponse inthe 1984 SIPP Panel. Proceedings of the American Statistical Association Section of SurveyResearch Methods. Arlington, VA: American Statistical Association, 296-301.
Sattler, J.M. (1992). Assessment of Children. 3rd ed. San Diego: Author.
Zahs, D., Pedlow, S., Morrissey, M., Marnell, P., and Nichols, B. (1995). High School and BeyondFourth Followup Methodology Report. NCES Publication No. 94-426. Washington, DC: U.S.Department of Education, National Center for Education Statistics.
Zill, N., Collins, M., West, J., and Hausken, E.G. (1995). Approaching Kindergarten: A Look atPreschoolers in the United States. NCES Publication No. 95-280. Washington, DC: U.S.Department of Education, National Center for Education Statistics.
