1 Cognitive Aspects Associated with Sample Selection Conducted by Respondents in Establishment Surveys La Toya Barnett Thomas Rebecca L. Morrison Grace E. ONeill U. S. Census Bureau 2 Outline Background Methodology Cognitive requirements of selecting a sample Results Conclusion Future Research 3 Background: CFS Information CFS = Commodity Flow Survey. Mandatory and collected every 5 years. Last survey covered 2002, next survey to cover Collects information on shipment of goods across the country for a 1 week reference period. Unique Feature: respondents required to select their own sample to complete form. 4 Background: The Issue Asking non-statisticians to perform a major statistical task systematic sampling. Not a normal survey task in general. Respondents drawing their own sample was an issue in 2002 survey. Sponsors suggested cognitive testing in preparation for Sample selection covered in 3 sections (Items D, E, and F) of the survey. 5 Background: The Issue (contd) Item D total number of outbound shipments during reference period. Item E guides Rs through process of selecting their sample, based on their response in Item D. Item F information about shipments from their sample (dates, type of shipment, destination, how transported). 6 Methodology Three rounds of cognitive interviews (October 2005 August 2006) Nationwide 64 interviews total Companies were a mix of industries, sizes, products, and shipping arrangements. 7 Methodology (contd) Tourangeaus (1984) Four Step Cognitive Response Model Comprehension Retrieval Judgment Communication 8 Cognitive Requirements of Selecting a Sample: Comprehension Understanding navigation and meaning of words and/or diagrams. Getting lost in navigation causes difficulty in completing task. Basically, if Rs lack comprehension, the task of selecting their own sample will be difficult. 9 Cognitive Requirements of Selecting a Sample: Retrieval Accessing several types of records: Fleet of vehicles and/or trucks Parcel delivery services Customer pick-ups Records could be on paper, in electronic form, or both. Records in multiple places need to know that they exist and how to access them. 10 Cognitive Requirements of Selecting a Sample: Judgment Which records Rs will select to report in survey. How to apply instructions to their records. How many records will be in my sample? How many do I need to report for? What if I cannot access all necessary records? 11 Results: Round 1 The 2002 form was used in this round of testing to identify shortcomings. CFS Questionnaire, Item E CFS Questionnaire, Item E CFS Questionnaire, Item E 15 Sample Selection vs. Audit Some Rs thought we were asking for an audit of shipping records and suggested calling Item E an Audit of Shipping Records Audit can have a negative connotation Item E could be similar to an audit, but is in fact systematic sampling. 16 Results: Round 2 Purpose of this round was to resolve issues from Round 1 Item E revised from Round 1: Placed on one page Eliminated diagrams and split table into two pieces Instructions turned into questions 17 Item E, Round 2 18 Results: Round 2 (contd) Having Item E on one page was helpful. The revised Item E did not adequately connect Items D and F. Asks Rs to re-report their response to Item D Later asked how many shipments they would report in Item F Were able to complete Item E, but did not know why 19 Results: Round 2 (contd) Language continued to be an issue Selection rate and selection method Difficult for people with non-statistical backgrounds to comprehend Rs who reported