Early Literacy Quick Assessment (ELQA) Technical Report ... · Both assessments measure alphabet...

1

Early Literacy Quick Assessment (ELQA) Technical Report: Validity and Reliability

Angela Urick

University of Oklahoma, College of Education

2

Early Literacy Quick Assessment (ELQA) Technical Report: Validity and Reliability

Statement of Purpose

The Educational Training, Evaluation, Assessment, and Measurement (E-TEAM) Department at the University of Oklahoma Outreach partnered with a third-party research methodologist to complete a validation study for the Early Literacy Quick Assessment (ELQA), an assessment tool developed by E-TEAM and the University of Oklahoma Outreach’s Center for Early Childhood Professional Development (CECPD). The assessment is designed to measure children’s knowledge of early literacy fundamentals. This tool can be used throughout the school year to aid teachers with large group, small group, and individualized instruction. The ELQA assesses four early literacy domains, or skill sets (alphabet knowledge, print concepts, phonological awareness, and vocabulary), through seven subtests:

• Uppercase Letter Identification • Lowercase Letter Identification • Letter Sounds • Print Concepts • Rhyming • Expressive Vocabulary • Receptive Vocabulary

This validation study is used to ensure that the ELQA is a valid and reliable tool for assessing early literacy skills. The study is also used to inform the ELQA developers of necessary changes to the existing assessment to increase its reliability and validity as an early literacy assessment tool.

Overview of Study

In an independent research study, the ELQA was statistically examined for its accuracy (validity) and consistency (reliability) in measuring early literacy skills. Out of the seven skills that ELQA measured, four (print concepts, upper case letters, lower case letters and letter sounds) were measured using the same items each time it was administered. Three skills (expressive vocabulary, receptive vocabulary and rhyming) have items that differ for each administration. Each unique administration for the seven skills across the five time periods was treated as an independent assessment.

This study focused mainly on construct validity and reliability to justify the extent that each administration of the ELQA accurately measures the early literacy skills intended. Construct validity and reliability results demonstrate whether or not items in the assessment need to be edited and/or provide evidence that the assessment is statistically sound. In addition, concurrent validity with similar assessments was examined to show the extent that the ELQA measures

3

similar concepts. With concurrent validity results, users can situate the content of the ELQA with other available assessments that are relevant to early literacy skills.

Data collected from Oklahoma pre-kindergarten students during the 2013-2014 school year were used to examine validity and reliability in this study. A sample of 201 students (see Table 1.1, sample A) was analyzed for construct validity and reliability. This group took the ELQA five times throughout the school year. A sample of 423 students (see Table 1.1, sample B) was analyzed for concurrent validity, which was assessed using scores from a final administration of the ELQA given at the end of the school year and scores from the Peabody Picture Vocabulary Test (PPVT-4; Dunn & Dunn, 2013) and the Phonological Awareness Literacy Screening, PreK (PALS-PreK, Invernizzi, Sullivan, Meier, & Swank, 2004) given at the end of the school year. The sample sizes were sufficient to examine the number of items, which range from 10 to 26 items, tested in each unique administration of the ELQA subtests (Cohen, 1992). The characteristics of each sample are reported to show the representativeness of the sample.

Concurrent Validity

The ELQA assesses a comprehensive range of skills compared to other relevant early literacy assessments. The PPVT-4 (Dunn & Dunn, 2013) is designed to measure receptive vocabulary through the respondent’s selection of a picture that illustrates a words meaning. The ELQA measures receptive vocabulary through one of its subtests. When related to PPVT-4, ELQA and PPVT-4 have a moderate, .459, correlation. If only the subtest of ELQA receptive vocabulary is correlated with the PPVT-4, which is a more relevant match of content, a moderate relationship remains, .499. The ELQA and the PALS-PreK (Invernizzi, Sullivan, Meier & Swank, 2004) have a wider range of skills included. Both assessments measure alphabet knowledge, letter sounds, rhyming and print concepts. ELQA incorporates these common concepts as well as vocabulary. The ELQA and PALS-PreK highly correlate at .919. If correlated by similar subtests, the relationships are moderate for written assessment items to measure print/word awareness (.427) and rhyming (.633), which shows that the construction and approach to these items is similar yet not highly aligned. As expected, the subtests which address alphabet knowledge – uppercase letters (.948), lowercase letters (.747), and letter sounds (.827) – had higher concurrent validity. While the alphabet items are the same across assessments, the overall order of the assessment, assessment and subtest length, and way in which the content is presented differs. See Tables 1.2, 1.3, and 1.4 for results.

4

Table 1.1 Student Demographics

Note: Percentages may not equal 100% due to rounding.

Table 1.2 Assessment Scores Examined for Concurrent Validity

Table 1.3 Concurrent Validity Correlations for Total Raw Scores

Assessment Correlation with ELQA

PALS .919 PPVT .459

Note: Significant at p<.01

Characteristic Sample A (n=201)

Sample B (n=423)

Sex Female 94 (47%) 205 (49%) Male 107 (53%) 218 (52%) Ethnicity African American 19 (9%) 22 (5%) American Indian 21 (10%) 62 (15%) Caucasian 141 (70%) 293 (69%) Hispanic 20 (10%) 42 (10%) Other 0 (0%) 4 (1%) Language Spoken at Home English 182 (91%) 390 (92%) Spanish 16 (8%) 29 (7%) Other 3 (1%) 4 (1%) Length of School Day Full Day 176 (88%) 370 (88%) Half Day 25 (12%) 53 (13%)

Variables N Mean Standard Deviation Min Max

Assessments PPVT raw score 423 85.06 19.81 13 129 PALS raw score 423 64.13 28.66 0 98 ELQA raw score 423 99.81 31.12 20 133

5

Table 1.4 Concurrent Validity Correlations for Relevant Subtests

Other Assessment Correlation ELQA Subtest PPVT raw score .499 Receptive

Vocabulary PALS print/word awareness .427 Print Concepts PALS uppercase letters .948 Uppercase Letters PALS lowercase letters .747 Lowercase Letters PALS letter sounds .827 Letter Sounds PALS rhyme awareness .633 Rhyming

Note: Significant at p<.01

6

Construct Validity

The ELQA measures seven different skills: four with static items and three with rotating items. Each skill was assessed five times throughout the school year for a total of 19 unique administrations. Each subtest has between 10 and 26 items. To test the seven concepts at each time period, participants responded to a total of 143 items.

The results of the construct validity analysis provide evidence that the 19 administrations appropriately measure one concept – in this case, early literacy fundamentals. Eigenvalues are reported to demonstrate the extent that a factor or concept is substantively represented in the data (Fabrigar, Wegener, MacCallum, & Strahan, 1999). A comparison of eigenvalues for one factor versus a second factor shows the degree to which only one factor or concept is measured compared to multiple concepts. The degree of decrease in eigenvalues between factor one and two shows that each ELQA subtest likely measures one substantive concept. Further, the results of the fit statistics also demonstrate the extent that the one main concept fits the data based on general, yet not absolute, cutoffs (Hu & Bentler, 1999; Steiger, 2007). In Table 2.2, each fit statistic column is labeled with a general cutoff for appropriate fit (e.g., CFI >.9; see Hu & Bentler, 1999). The nine administrations appear to likely test one skill when construct validity is examined. See Table 2.1 for sample characteristics and Table 2.2 for results.

7

Table 2.1 Sample Characteristics for Construct Validity

Assessment N

Average Age in Months % Female

% Non-White

% Language Other than English

Static Items1 Print Concepts 274 55.25 47% 29% 9% Upper Case Letters 274 55.25 47% 29% 9% Lower Case Letters 216 55.66 50% 27% 7% Letter Sounds 213 55.56 51% 27% 7% Rotating Items2 Expressive Vocabulary Period 1 279 54.62 47% 31% 9% Period 2 257 55.25 46% 29% 9% Period 3 226 55.18 48% 30% 10% Period 4 232 55.47 47% 28% 9% Period 5 243 55.19 46% 30% 9% Receptive Vocabulary 202 55.64 49% 30% 10% Rhyming Period 1 276 54.66 47% 31% 9% Period 2 257 55.25 46% 29% 9% Period 3 226 55.18 48% 30% 10% Period 4 231 55.55 48% 28% 9% Period 5 243 55.19 46% 30% 9%

Note: 1Assessments with static items across periods were represented by a sample of students who participated in the end of year administration. 2Assessments with rotating items have a unique set of items for each time period so each administration was treated as an individual assessment.

8

Table 2.2 Results for Construct Validity

Eigenvalues Fit Statistics for One Factor

Assessment Factor 1 Factor 2

CFI ( > .9)

TLI (>.9)

RMSEA (< .06)

Static Items Print Concepts 6.720 0.894 .991 .988 .037 Upper Case Letters 20.958 0.763 .998 .998 .020 Lower Case Letters 13.983 2.415 .962 .959 .030 Letter Sounds 17.925 1.464 .996 .995 .023 Rotating Items Expressive Vocabulary Period 1 13.912 1.906 .974 .971 .030 Period 2 13.092 1.828 .986 .985 .024 Period 3 14.648 1.591 .997 .997 .012 Period 4 15.224 2.302 .985 .984 .024 Period 5 14.701 2.257 .996 .996 .010 Receptive Vocabulary Period 1 10.767 1.753 .981 .979 .025 Period 2 9.971 2.740 .961 .956 .031 Period 3 8.398 1.734 .971 .967 .028 Period 4 9.810 2.707 .968 .964 .029 Period 5 9.040 2.188 .965 .960 .031 Rhyming Period 1 5.915 1.616 .969 .960 .143 Period 2 6.376 1.268 .973 .966 .103 Period 3 6.603 1.388 .993 .991 .087 Period 4 6.682 1.154 .989 .985 .099 Period 5 7.157 1.071 .995 .994 .069

9

Reliability

The internal consistency of each ELQA subtest was examined using Cronbach’s alpha level and Guttman split-half reliability. Cronbach’s alpha is an index of reliability which measures the extent that items in an assessment are related one another or the same construct (Cronbach, 1951; Henson, 2001). Like a correlation, both Cronbach’s alpha and Guttman split-half range between 0 and 1, with 1 indicating perfect reliability. Guttman split-half examines internal consistency by dividing items in half to test reliability in and between groups of items as an additional measure of consistency (Guttman, 1945; Robinson, Shaver & Wrightsman, 1991). See Table 2.1 for sample characteristics and Table 3.1 for results.

Item Validity and Reliability

The examination of construct validity and reliability of each assessment also yields evidence of the degree that each item relates to the total score for that subtest. Factor loadings measure how much each item loads onto the common concept or factor (Fabrigar, Wegener, MacCallum, & Strahan, 1999). Similarly, item-to-total correlations represent the extent that each item correlates with the total subtest score but does not test for multiple concepts or factors (Gerbing & Anderson, 1988). Factor loading and item-to-total correlations range between 0 and 1, with 1 indicating a perfect relationship. Most items load and correlate with the tested concept. Refer to Tables 4.1 to 4.19 for item results by subtest.

10

Table 3.1 Results for Reliability

Assessment Cronbach’s Alpha

Guttman Split-Half

Static Items Print Concepts .834 .798 Upper Case Letters .967 .950 Lower Case Letters .900 .885 Letter Sounds .947 .948 Rotating Items Expressive Vocabulary Period 1 .884 .879 Period 2 .865 .882 Period 3 .887 .909 Period 4 .870 .865 Period 5 .881 .887 Receptive Vocabulary Period 1 .834 .775 Period 2 .810 .802 Period 3 .797 .783 Period 4 .818 .803 Period 5 .786 .763 Rhyming Period 1 .816 .7701 Period 2 .847 .8571 Period 3 .879 .8711 Period 4 .876 .9141 Period 5 .891 .9091

Note: 1Items split with both question stems represented in each half.

11

Table 4.1 Item Results for Print Concepts

Print Concepts

Factor Loading (>.4)

Item-to-Total Correlation

Point to the name, or title, of the story (A1Q1149) .885 .568 Point to where I start to read the story (A1Q1150) .831 .556 Which way do I go when I read (A1Q1151) .918 .584 Where do I go after that [end of line] (A1Q1152) .898 .631 Point to each word as I read (A1Q1153) .764 .516 What is this for [period] (A1Q1154) .936 .706 Point to a capital letter (A1Q1155) .605 .391 Point to a letter (A1Q1156) .585 .342 Point to a word (A1Q1157) .808 .535 Point to the first letter in a word (A1Q1158) .814 .581

12

Table 4.2 Item Results for Upper Case Letters

Upper Case Letters Factor Loading Item-to-Total Correlation

A (A4Q1083) .906 .665 B (A4Q10840 .838 .619 C (A4Q1085) .922 .726 D (A4Q1086) .867 .690 E (A4Q1087) .962 .812 F (A4Q1088) .956 .821 G (A4Q1089) .885 .714 H (A4Q1090) .875 .709 I (A4Q1091) .936 .799 J (A4Q1092) .956 .798 K (A4Q1093) .904 .736 L (A4Q1094) .903 .717 M (A4Q1095) .917 .761 N (A4Q1096) .911 .748 O (A4Q1097) .937 .627 P (A4Q1098) .933 .787 Q (A4Q1099) .855 .685 R (A4Q1100) .916 .752 S (A4Q1101) .895 .702 T (A4Q1102) .898 .737 U (A4Q1103) .870 .703 V (A4Q1104) .892 .727 W (A4Q1105) .790 .611 X (A4Q1106) .944 .554 Y (A4Q1107) .877 .708 Z (A4Q1108) .897 .706

13

Table 4.3 Item Results for Lower Case Letters

Lower Case Letters Factor Loading Item-to-Total Correlation

a (A6Q1391) .766 .495 b (A6Q1392) .483 .322 c (A6Q1393) .816 .415 d (A6Q1394) .806 .545 e (A6Q1395) .871 .544 f (A6Q1396) .814 .576 g (A6Q1397) .821 .619 h (A6Q1398) .768 .567 i (A6Q1399) .665 .412 j (A6Q1400) .671 .442 k (A6Q1401) .799 .451 l (A6Q1402) .712 .526 m (A6Q1403) .759 .533 n (A6Q1404) .829 .611 o (A6Q1405) .617 .323 p (A6Q1406) .758 .515 q (A6Q1407) .819 .610 r (A6Q1408) .808 .502 s (A6Q1409) .706 .302 t (A6Q1410) .771 .566 u (A6Q1411) .824 .619 v (A6Q1412) .833 .622 w (A6Q1413) .738 .461 x (A6Q1414) .617 .323 y (A6Q1415) .592 .373 z (A6Q1416) .789 .383

14

Table 4.4 Item Results for Letter Sounds

Letter Sounds Factor Loading Item-to-Total Correlation

A (A7Q1499) .706 .470 B (A7Q1500) .768 .460 C (A7Q1501) .832 .649 D (A7Q1502) .831 .634 E (A7Q1503) .794 .613 F (A7Q1504) .888 .663 G (A7Q1505) .876 .704 H (A7Q1506) .899 .709 I (A7Q1507) .860 .658 J (A7Q1508) .830 .594 K (A7Q1509) .870 .620 L (A7Q1510) .745 .544 M (A7Q1511) .856 .681 N (A7Q1512) .841 .643 O (A7Q1513) .801 .604 P (A7Q1514) .776 .516 Q (A7Q1515) .864 .687 R (A7Q1516) .914 .745 S (A7Q1517) .752 .355 T (A7Q1518) .893 .651 U (A7Q1519) .894 .697 V (A7Q1520) .807 .624 W (A7Q1521) .949 .781 X (A7Q1522) .808 .620 Y (A7Q1523) .886 .672 Z (A7Q1524) .770 .513

15

Table 4.5 Item Results for Expressive Vocabulary Period 1

Expressive Vocabulary, Period 1 Factor Loading


What is this? Bat (A2Q180) .721 .504 What is this? Bird (A2Q213) .789 .479 What is this? Blocks (A2Q214) .699 .454 What is this? Cookie (A2Q215) .569 .256 What is this? Doctor (A2Q216) .540 .362 What is this? Dress (A2Q217) .816 .565 What is this? Eyes (A2Q218) .775 .486 What is this? Flag (A2Q219) .861 .620 What is this? Fox (A2Q220) .588 .351 What is this? Glue (A2Q221) .823 .582 What is this? Grapes (A2Q222) .652 .448 What is this? Hand (A2Q223) .861 .497 What is this? Horse (A2Q224) .948 .624 What is this? Kangaroo (A2Q225) .713 .489 What is this? Nest (A2Q226) .744 .508 What is this? Nose (A2Q227) .617 .366 What is this? Pumpkin (A2Q228) .766 .566 What is this? Rabbit (A2Q229) .764 .525 What is this? Taxi (A2Q230) .239 .169 What is this? Television (A2Q231) .988 .620 What is this? Toothbrush (A2Q232) .930 .671 What is this? Zebra (A2Q233) .593 .412 What is this? Rope [2] (A2Q234) .656 .463 What is this? Cell phone (A2Q235) .811 .402 What is this? Feather (A2Q236) .842 .605

16




What is this? Airplane (A2Q237) .934 .596 What is this? Banana (A2Q238) .539 .254 What is this? Bike (A2Q239) .950 .623 What is this? Broccoli (A2Q240) .538 .383 What is this? Corn (A2Q241) .884 .666 What is this? Fireman (A2Q242) .842 .611 What is this? Guitar (A2Q243) .615 .383 What is this? Hippopotamus (A2Q244) .722 .455 What is this? Key (A2Q245) .861 .470 What is this? Monkey (A2Q246) .416 .229 What is this? Nail (A2Q247) .523 .347 What is this? Plate (A2Q248) .850 .544 What is this? Refrigerator (A2Q249) .921 .636 What is this? Saw (A2Q250) .706 .406 What is this? Sheep (A2Q251) .689 .488 What is this? Socks (A2Q252) .932 .450 What is this? Squirrel (A2Q253) .742 .532 What is this? Sun (A2Q254) .885 .499 What is this? Telephone (A2Q255) .610 .335 What is this? Umbrella (A2Q256) .625 .426 What is this? Yoyo (A2Q257) .324 .216 What is this? Jump rope (A2Q258) .793 .573 What is this? Van (A2Q259) .451 .301 What is this? Tiger (A2Q260) .491 .331 What is this? Lips (A2Q261) .869 .504

17




What is this? Bell (A2Q181) .853 .608 What is this? Boat (A2Q182) .771 .451 What is this? Book (A2Q183) .745 .322 What is this? Bread (A2Q184) .902 .665 What is this? Camel (A2Q185) .624 .407 What is this? Chain (A2Q186) .718 .454 What is this? Chair (A2Q187) .842 .445 What is this? Clock (A2Q188) .746 .352 What is this? Couch (A2Q189) .814 .557 What is this? Duck (A2Q190) .723 .452 What is this? Giraffe (A2Q191) .664 .457 What is this? Hammer (A2Q192)1 --- .720 What is this? Hot dog (A2Q194) .989 .622 What is this? Leaf (A2Q195) .959 .642 What is this? Orange (A2Q196) .687 .414 What is this? Potato (A2Q197) .671 .433 What is this? Tomato (A2Q198) .496 .336 What is this? Wagon (A2Q199) .783 .579 What is this? Window (A2Q200) .622 .417 What is this? Worm (A2Q201) .685 .485 What is this? Teeth (A2Q202) .994 .657 What is this? Mailman (A2Q206) .856 .570 What is this? Snake (A2Q207) .766 .414 What is this? Eyeglasses (A2Q208) .980 .621 What is this? Box (A2Q1311) .855 .445

Note: 1Removed from factor analysis due to negative residual variance

18




What is this? Cake (A2Q262) .911 .566 What is this? Carrot (A2Q263) .693 .336 What is this? Clouds (A2Q264) .615 .366 What is this? Ear (A2Q265) .887 .411 What is this? Fork (A2Q266) .893 .632 What is this? Frog (A2Q267) .962 .599 What is this? Hat (A2Q268) .919 .514 What is this? Kite (A2Q269) .914 .621 What is this? Lock (A2Q270) .696 .427 What is this? Mailbox (A2Q271) .792 .558 What is this? Map (A2Q272) .434 .252 What is this? Money (A2Q273) .455 .175 What is this? Motorcycle (A2Q274) .742 .460 What is this? Pig (A2Q275) .982 .590 What is this? Rock (A2Q276) .976 .537 What is this? Shoe (A2Q277) .834 .370 What is this? Slide (A2Q278) .991 .680 What is this? Spoon (A2Q279) .970 .662 What is this? Table (A2Q280) .662 .428 What is this? Tent (A2Q281) .853 .572 What is this? Watermelon (A2Q282) .739 .428 What is this? Whale (A2Q283) .490 .305 What is this? Bear photo (A2Q284) .885 .492 What is this? Tractor (A2Q285) .733 .507 What is this? King (A2Q286) .693 .461

19




What is this? Ant (A2Q287) .834 .598 What is this? Ball (A2Q288) .455 .152 What is this? Bed (A2Q289) .919 .492 What is this? Coat (A2Q290) .518 .292 What is this? Cup (A2Q291) .715 .430 What is this? Dog (A2Q292) .741 .209 What is this? Finger (A2Q293) .831 .442 What is this? Firetruck (A2Q294) .790 .537 What is this? Goat (A2Q295) .554 .343 What is this? Ice cream (A2Q296) .821 .439 What is this? Ladder (A2Q297) .883 .664 What is this? Ladybug (A2Q298) .912 .609 What is this? Lion (A2Q299) .553 .319 What is this? Milk (A2Q300) .665 .362 What is this? Mouse (A2Q301) .939 .702 What is this? Owl (A2Q302) .919 .712 What is this? Policeman (A2Q303) .629 .318 What is this? Rainbow (A2Q304) .842 .367 What is this? Sled (A2Q305) .843 .576 What is this? Star (A2Q306) .804 .352 What is this? Stop sign (A2Q307) .873 .581 What is this? Swing (A2Q308) .782 .469 What is this? Truck (A2Q309) .644 .394 What is this? Paintbrush (A2Q310) .894 .676 What is this? Foot (A2Q311) .839 .526

20

Table 4.10 Item Results for Receptive Vocabulary Period 1

Receptive Vocabulary, Period 1 Factor Loading


Point to the shoulder (A5Q1229) .526 .370 Point to the knee (A5Q1249) .909 .432 Point to the shirt (A5Q1238) .751 .385 Point to the pants (A5Q1248) .533 .199 Point to the store (A5Q1214) .597 .375 Point to the home (A5Q1224) .778 .518 Point to the picture that shows under (A5Q1240) .756 .467 Point to the picture that shows around (A5Q1250) .387 .197 Point to the picture that is a sad face (A5Q1227) .868 .483 Point to the picture that is a happy face (A5Q1237) .933 .529 Point to the forest (A5Q1216) .865 .583 Point to the desert (A5Q1226) .735 .506 Point to the seed (A5Q1215) .815 .588 Point to the petal (A5Q1235) .375 .246 Point to the child who is third in line (A5Q1242) .675 .410 Point to the child who is last in line (A5Q1222) .779 .511 Point to the circle (A5Q1211) .862 .503 Point to the triangle (A5Q1221) .724 .420 Point to the helicopter (A5Q1213) .912 .541 Point to the tractor (A5Q1223) .781 .421

21




Point to the elbow (A5Q1239) .817 .420 Point to the ankle (A5Q1209) .242 .204 Point to the pants (A5Q1248) .830 .255 Point to the scarf (A5Q1228) .936 .535 Point to the fire station (A5Q1244) .598 .196 Point to the school (A5Q1254) .629 .207 Point to the picture that shows through (A5Q1220) .520 .362 Point to the picture that shows over (A5Q1230) .649 .435 Point to the picture that is an angry face (A5Q1217) .886 .600 Point to the picture that is a sad face (A5Q1227) .800 .427 Point to the ocean (A5Q1236) .888 .568 Point to the cloud (A5Q1246) .876 .507 Point to the branch (A5Q1255) .672 .452 Point to the seed (A5Q1215) .883 .608 Point to the child who is second in line (A5Q1232) .534 .352 Point to the child who is third in line (A5Q1242) .585 .370 Point to the square (A5Q1231) .693 .426 Point to the line (A5Q1241) .724 .291 Point to the trailer (A5Q1233) .549 .378 Point to the propeller (A5Q1243) .601 .414

22




Point to the wrist (A5Q1219) .410 .236 Point to the shoulder (A5Q1229) .572 .403 Point to the scarf (A5Q1228) .947 .493 Point to the shirt (A5Q1238) .838 .385 Point to the home (A5Q1224) .780 .487 Point to the library (A5Q1234) .669 .271 Point to the picture that shows behind (A5Q1210) .417 .248 Point to the picture that shows around (A5Q1250) .443 .226 Point to the picture that is a happy face (A5Q1237)1 --- .499 Point to the picture that is a thankful face (A5Q1247) .608 .406 Point to the desert (A5Q1226) .748 .465 Point to the ocean (A5Q1236) .756 .433 Point to the petal (A5Q1235) .436 .316 Point to the branch (A5Q1255) .649 .450 Point to the child who is last in line (A5Q1222) .712 .484 Point to the child who is second in line (A5Q1232) .663 .399 Point to the rectangle (A5Q1251) .624 .412 Point to the circle (A5Q1211) .846 .482 Point to the caboose (A5Q1253) .391 .265 Point to the helicopter (A5Q1213) .805 .435

Note: 1Removed from factor analysis due to negative residual variance

23




Point to the elbow (A5Q1239) .756 .369 Point to the ankle (A5Q1209) .333 .239 Point to the pants (A5Q1248) .505 .138 Point to the shirt (A5Q1238) .819 .381 Point to the store (A5Q1214) .633 .431 Point to the school (A5Q1254) .537 .123 Point to the picture that shows under (A5Q1240) .788 .474 Point to the picture that shows through (A5Q1220) .507 .350 Point to the picture that is a happy face (A5Q1237) .911 .481 Point to the picture that is an angry face (pA5Q1217) .858 .614 Point to the moon (A5Q1256) .691 .200 Point to the forest (A5Q1216) .804 .549 Point to the seed (A5Q1215) .831 .587 Point to the branch (A5Q1255) .815 .478 Point to the child who is second in line (A5Q1232) .723 .384 Point to the child who is last in line (A5Q1222) .645 .505 Point to the triangle (A5Q1221) .655 .404 Point to the square (A5Q1231) .673 .406 Point to the tractor (A5Q1223) .930 .442 Point to the trailer (A5Q1233) .563 .393

24




Point to the knee (A5Q1249) .933 .410 Point to the wrist (A5Q1219) .300 .175 Point to the scarf (A5Q1228) .971 .528 Point to the pants (A5Q1248) .795 .231 Point to the library (A5Q1234) .588 .225 Point to the fire station (A5Q1244) .532 .187 Point to the picture that shows over (A5Q1230) .644 .444 Point to the picture that shows behind (A5Q1210) .384 .228 Point to the picture that is a thankful face (A5Q1247) .604 .402 Point to the picture that is an angry face (A5Q1217) .859 .548 Point to the desert (A5Q1226) .728 .441 Point to the ocean (A5Q1236) .876 .485 Point to the petal (A5Q1235) .433 .316 Point to the seed (A5Q1215) .856 .581 Point to the child who is third in line (A5Q1242) .644 .469 Point to the child who is second in line (A5Q1232) .567 .384 Point to the line (A5Q1241) .799 .279 Point to the rectangle (A5Q1251) .656 .434 Point to the propeller (A5Q1243) .567 .373 Point to the caboose (A5Q1253) .396 .283

Table 4.15 Item Results for Rhyming Period 1

Rhyming, Period 1 Factor Loading


Do these words rhyme? Ball- tall (A3Q1159) .381 .232 Do these words rhyme? Fold- kind (A3Q1160) .628 .420 Do these words rhyme? Pen- cup (A3Q1161) .555 .339 Do these words rhyme? Hat-bat (A3Q1162) .615 .337 Do these words rhyme? Sand- hand (A3Q1163) .636 .363 Tell me a word that rhymes with do (A3Q1184) .940 .672 Tell me a word that rhymes with head (A3Q1185) .931 .683 Tell me a word that rhymes with fun (A3Q1186) .957 .667 Tell me a word that rhymes with got (A3Q1187) .976 .701 Tell me a word that rhymes with mill (A3Q1188) .976 .717

25




Do these words rhyme? Arm- fork (A3Q1164) .590 .394 Do these words rhyme? Rug- bug (A3Q1165) .692 .394 Do these words rhyme? Sip- tip (A3Q1166) .571 .343 Do these words rhyme? Jam- rain (A3Q1167) .619 .441 Do these words rhyme? Fox- box (A3Q1168) .804 .421 Tell me a word that rhymes with rug (A3Q1189) .845 .625 Tell me a word that rhymes with be (A3Q1190) .863 .638 Tell me a word that rhymes with go (A3Q1191) .922 .701 Tell me a word that rhymes with sit (A3Q1192) .955 .729 Tell me a word that rhymes with pan (A3Q1193) .962 .742




Do these words rhyme? Map- tap (A3Q1169) .445 .267 Do these words rhyme? Rose- hose (A3Q1170) .664 .391 Do these words rhyme? Task- pay (A3Q1171) .768 .571 Do these words rhyme? pane- gain (A3Q1172) .504 .312 Do these words rhyme? girl- bug (A3Q1173) .686 .461 Tell me a word that rhymes with tie (A3Q1194) .936 .773 Tell me a word that rhymes with day (A3Q1195) .977 .823 Tell me a word that rhymes with big (A3Q1196) .979 .804 Tell me a word that rhymes with bell (A3Q1197) .970 .802 Tell me a word that rhymes with men (A3Q1198) .934 .745

26




Do these words rhyme? dig- rig (A3Q1174) .628 .428 Do these words rhyme? word- back (A3Q1175) .694 .479 Do these words rhyme? bed- red (A3Q1176) .725 .472 Do these words rhyme? best- vest (A3Q1177) .529 .323 Do these words rhyme? lip- rug (A3Q1178) .670 .456 Tell me a word that rhymes with take (A3Q1199) .929 .751 Tell me a word that rhymes with ball (A3Q1200) .988 .787 Tell me a word that rhymes with ran (A3Q1201) .969 .797 Tell me a word that rhymes with hat (A3Q1202) .956 .709 Tell me a word that rhymes with run (A3Q1203) .936 .721




Do these words rhyme? ball- tall (A3Q1179) .667 .423 Do these words rhyme? fold- kind (A3Q1180) .742 .505 Do these words rhyme? pen- cup (A3Q1181) .795 .561 Do these words rhyme? hat- bat (A3Q1182) .698 .424 Do these words rhyme? sand- hand (A3Q1183) .649 .341 Tell me a word that rhymes with do (A3Q1204) .925 .765 Tell me a word that rhymes with head (A3Q1205) .984 .828 Tell me a word that rhymes with fun (A3Q1206) .985 .842 Tell me a word that rhymes with got (A3Q1207) .920 .725 Tell me a word that rhymes with mill (A3Q1208) .972 .799

27

Conclusions and Recommendations

The ELQA has acceptable validity and reliability based on the above analyses and sample. There was an appropriate level of concurrent validity with two relevant assessments, the PALS-PreK and the PPVT-4. The results of the construct validity analysis demonstrate that each administration of the subtests likely measured one concept, as anticipated. Reliability estimates ranged from moderate to high, beyond levels of acceptability. Most individual items loaded onto the concept tested and had an acceptable degree of reliability when correlated with the other items in the subtest.

While the ELQA is considered a sound assessment for early literacy skills, some results indicate areas for improvement or for further study. In the analysis of concurrent validity, the relationship between the PPVT-4 total scores and the ELQA receptive vocabulary subtest (.499), and the PALS-PreK print/word awareness and the ELQA print concepts subtest (.427), showed a moderate degree of overlap when the concepts tested are similar. A more in-depth comparison of test content and presentation would explain the difference in how each assessment approaches the measurement of these like concepts. An understanding of these similarities and differences would lead to further conclusions about what the ELQA measures or does not measure for each of these concepts, which would provide more specific descriptions of the ELQA’s benefits and/or potential for item improvement.

Construct validity results demonstrated that the rhyming subtest could be improved. One fit statistic, which helps estimate the accuracy of testing one concept as opposed to multiple concepts, did not meet general cutoffs (see Table 2.2, RMSEA) for the rhyming subtest in all five administrations. This one fit statistic improved with the fit of two concepts/factors rather than a single concept to the assessment (results from two-factor model not reported above). Few items with the stem “do these words rhyme” loaded onto a second concept (Period 1: ball-tall, hat-bat, sand-hand; Period 2: arm-fork, jam-rain; Period 3: map-tap; Period 4: word-back, best-vest, lip-rug; Period 5: fold-kind, pen-cup). With the exception of Period 3, the analysis appeared to separate the “do these words rhyme” items into questions with rhyming words provided and questions without rhyming words provided. While a two-factor or concept model was a potential fit for the rhyming subtest, the one-factor model had similar fit as demonstrated by fit statistics other than the RMSEA estimate and moderate to high factor loadings for all items, as well as moderate to high reliability.

Upon interpreting results by item, a few questions had a lower degree of grouping with the corresponding subtest. For example, the following items had low factor loadings and item-to-total correlations: Expressive Vocabulary Period 1 “What is this? Taxi,” Expressive Vocabulary Period 2 “What is this? Yo-yo,” Receptive Vocabulary Period 1 “Point to the picture that shows ‘around’,” Receptive Vocabulary Period 1 “Point to the petal,” Receptive Vocabulary Period 2 “Point to the ankle,” Receptive Vocabulary Period 3 “Point to the caboose,” Receptive Vocabulary Period 4 “Point to the ankle,” Receptive Vocabulary Period 5 “Point to the wrist,”

28

Receptive Vocabulary Period 5 “Point to the picture that shows ‘behind’,” Receptive Vocabulary Period 5 “Point to the caboose,” Rhyming Period 1 “Do these words rhyme? Ball-tall.” Many possible reasons exist for lower item grouping estimates. In addition, there is a need for balance between content validity and statistical validity and reliability scores. Further study of participant responses to items, rather than exploration of correct/incorrect scores alone, might provide more information about how to edit the content and presentation of these items to better fit with the subtest and align with content validity.

This initial validity and reliability study was necessary in order to provide a baseline as to the current accuracy and consistency of items included in the ELQA. Based on the recommendations above, all subtests and items can be finalized with any other preferred improvements. Once subtests/items have been edited, piloted, and finalized, a follow-up study of the ELQA with a representative sample and final advanced statistical techniques will complete assessment construction. A sample representative of the population of potential assessment participants will remove any sample bias, and scores can be standardized. Finally, Item Response Theory (IRT) models can be used to place different items/subtests in an assessment on a common scale. The benefit of IRT is to link multiple subtests and periods to a scale with the same meaning and takes into account ability level for each examinee to create cut-scores or proficiency standards.

29

References

Cohen, J. (1992). A power primer. Quantitative Methods in Psychology, 112(1), 155-159.

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Dunn, L., & Dunn, D. (2013). Technical Report: PPVT 4 Peabody Picture Vocabulary Test Fourth Edition. Pearson.

Fabrigar, L., Wegener, D., MacCallum, R., & Strahan, E. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299.

Gerbing, D., & Anderson, J. (1988). An updated paradigm for scale development incorporating unidimensionality and its assessment. Journal of Marketing, 25(2), 186-192.

Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282.

Henson, R. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development. 34(3), 177.

Hu, L., & Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.

Invernizzi, M., Sullivan, A., Meier, J., & Swank, L. (2004). PreK Teacher’s Manual: PALS Phonological Awareness Literacy Screening. Virginia Department of Education and University of Virginia, Curry School of Education.

Robinson, J., Shaver, P., and Wrightsman, L. (1991). Criteria for scale selection and evaluation. In Measures of Personality and Social Psycholo

Steiger, J. (2007). Understanding the limitations of global fit assessment in structural equation modeling. Personality and Individual Differences, 42(5), 893-898.

Date post:	06-Nov-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Early Literacy Quick Assessment (ELQA) Technical Report ... · Both assessments measure alphabet...

Documents