CONSTRUCTION AND VALIDATION OF A PROGRAMMED INSTRUCTION

BOOKLET: METHODS OF ADULT EDUCATION

by

DIANE ELMIRA BETTS B.N. U n i v e r s i t y of New Brunswick, 1968

A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF

MUSTER OFFSETS

i n the F a c u l t y

of

Education

We accept t h i s t h e s i s as conforming to the requ i r e d standard

THE UNIVERSITY OF BRITISH COLUMBIA

A p r i l , 1975

In presenting t h i s t h e s i s in p a r t i a l f u l f i l m e n t of the requirements f o r

an advanced degree at the U n i v e r s i t y of B r i t i s h Columbia, I agree that

the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r reference and study.

I f u r t h e r agree that permission for extensive copying of t h i s t h e s i s

f o r s c h o l a r l y purposes may be granted by the Head of my Department or

by h i s r e p r e s e n t a t i v e s . It i s understood that copying or p u b l i c a t i o n

of t h i s t h e s i s f o r f i n a n c i a l gain s h a l l not be allowed without my

w r i t t e n permission.

Department of /4~DUkT P-DOe/s-T-lorJ

The U n i v e r s i t y of B r i t i s h Columbia Vancouver 8, Canada

Date A-pfti^- 3 o ; qq s

ABSTRACT

The purpose of t h i s t h e s i s was to construct and v a l i d a t e a

programmed i n s t r u c t i o n booklet f o r the use of u n i v e r s i t y students

studying to become a d u l t educators. The concept of methods of a d u l t

education and the c l a s s i f i c a t i o n scheme f o r methods as devised by

Dr. C. Verner were used as course content. The branching or i n t r i n s i c

model was s e l e c t e d and a c r i t e r i o n t e s t and frames were constructed.

The booklet was e m p i r i c a l l y t e s t e d through a developmental

t e s t and two f i e l d t e s t s . Sixty-two students e n r o l l e d i n three c l a s s e s

of Education 412, during the 1974 summer and 1975 wi n t e r sessions a t

The U n i v e r s i t y of B r i t i s h Columbia, p a r t i c i p a t e d i n the v a l i d a t i o n

procedure. The developmental t e s t group c o n s i s t e d of twelve students

who sat w i t h the programmer, one at a time, as they s t u d i e d the booklet

and completed a pre and post c r i t e r i o n t e s t .

The f i e l d t e s t was designed as a course take home assignment

and students were r e s p o n s i b l e f o r r e t u r n i n g w r i t t e n answers to a pre

and post c r i t e r i o n t e s t . Ten students volunteered to study the booklet

during the f i r s t f i e l d t e s t . The second f i e l d t e s t assigned twenty

students to a pre and post c r i t e r i o n t e s t only c o n t r o l group design

and twenty students to an experimental group design who s t u d i e d the

booklet and completed a pre and post c r i t e r i o n t e s t .

A n a l y s i s of the data f o l l o w i n g the developmental and f i r s t f i e l d

t e s t s revealed many areas of weakness i n the booklet frames and i n the

i i

c r i t e r i o n t e s t . Consequently many major r e v i s i o n s were made to both.

Examination of the data f o l l o w i n g the second f i e l d t e s t i n d i c a t e d the

booklet met the standards of an e f f e c t i v e teaching program.

i i i

TABLE OF CONTENTS

Page

ABSTRACT i

LIST OF TABLES v

LIST OF FIGURES v i

ACKNOWLEDGMENTS ' . v i i

Chapter

I INTRODUCTION . 1

BACKGROUND OF THE PROBLEM 1

THE PROBLEM 3

Importance 'for Adult Education 3

DEFINITION OF TERMS 5

PLAN OF THE THESIS 6

I I REVIEW OF THE LITERATURE 7

PROGRAMMED VS. CONVENTIONAL INSTRUCTION 8

BRANCHING VS. LINEAR PARADIGMS 13

VARIABLES IN PROGRAM CONSTRUCTION 17

Overt vs. Covert Responses 18

Large Step vs. Small Step 21

I I I PROCEDURE 24

CONSTRUCTION OF THE BOOKLET 24

Subject S e l e c t i o n 24

Paradigm S e l e c t i o n 25

Content Development 30

VALIDATION OF THE BOOKLET 33

i v

Chapter Page

DsDevelopmenfLaleTest 34

FiEiel'deTests 36

DaDatanAftalysis 39

IV ANALYSIS OF THE DATA 41

CRITERION TEST 41

DEVELOPMENTAL TEST DATA 44

FIRST FIELD TEST DATA 52

SECOND FIELD TEST DATA 55

V SUMMARY, IMPLICATIONS FOR PRACTICE AND RESEARCH,

AND CONCLUSIONS 60

BIBLIOGRAPHY 68

APPENDIXES

A Programmed I n s t r u c t i o n a l Booklet: Methods of Adul t

Education 72

B V a l i d a t i o n Statement I5;2j

C Sample V a l i d a t i o n Progress Charts 16>

D Sample F i r s t C r i t e r i o n Test Wfij

E Sample Revised C r i t e r i o n Test 1^3

V

LIST OF TABLES

Table Page

1 Comparison of Programmed Inst r u c t i o n and Conventional Ins t r u c t i o n 10

2 The Results of 112 Studies Comparing Programmed

Ins t r u c t i o n with Conventional I n s t r u c t i o n 12

3 Summary of Results of T-test for C r i t e r i o n Test . . . . 42

4 Summary of Results using the Kuder Richardson Formula

20 . 4 43 5 Item D i f f i c u l t y Index for Mult i p l e Choice Questions . . 45

6 Results of Test Scores, Developmental Test Group . . . . 48

7 Results of T-test f o r C r i t e r i o n Test Scores, Developmental Test Group 48

8 Results of C r i t e r i o n Test Scores, F i r s t F i e l d Test Group 52

9 Results of T-test f o r C r i t e r i o n Test Scores, F i r s t F i e l d Test Group 53

10 Results of C r i t e r i o n Test f o r Control and Experimental Groups 57

11 Summary of Results of Anovar 57

12 Summary of C r i t e r i o n Test ScoresExperimental Group . . 58

v i

LIST OF FIGURES

Figure Page

I Interrelationships of the Steps i n Programming 30

II The Programming Process 31

I I I The Validation Process 35

IV Experimental Design for Second F i e l d Test 37

V lfeMiefecdforSetfsr.fofoStM'edestandn6rStiies.eFol

v i i

ACKNOWLEDGMENTS

I wish to give thanks to Dr. James Thortifconfiorhhisaadvice,

patience and a s s i s t a n c e during the c o n s t r u c t i o n and v a l i d a t i o n of the

bookl e t , Methods of Adult Education, and f o r h i s guidance during the

w r i t i n g of t h i s t h e s i s . In a d d i t i o n I have appreciated the i n t e r e s t

and encouragement of Dr. C. Verner, Dr. G. Dickinson, and Dr. S.

Blank. The a s s i s t a n c e of A. Blunt w i t h the s t a t i s t i c a l data was an

i n v a l u a b l e a i d .

So many of my f e l l o w students i n a d u l t education took time

to o f f e r a s s i s t a n c e , complete the c r i t e r i o n t e s t s and/or p a r t i c i p a t e

i n the v a l i d a t i o n procedure, and to each of them I am most g r a t e f u l .

I am als o indebted to the many people who helped w i t h the t y p i n g of

the many suc c e s s i v e d r a f t s o f the booklet and the t h e s i s , e s p e c i a l l y

to Mrs. C a r o l Moreno and Mrs. L. Cochrane.

And f i n a l l y , my very s p e c i a l thanks to my husband Gord, w i t h -

out whose constant support and encouragement t h i s t h e s i s would not

have been completed.

CHAPTER I

INTRODUCTION

Background of the Problem

The exponential r a t e at which new knowledge i s being generated

i s unique to our modern s o c i e t y , and i s of such magnitude that i t

a f f e c t s every person and has an impact on every area of human l i f e .

"The r a t e at which man acquires knowledge doubled the f i r s t time around

1700, and again around 1900. The t h i r d doubling occurred i n 1950 and

the f o u r t h i n 1960" (3:181). The knowledge to be gathered and dissemin-

ated i s i n c r e a s i n g at such an astounding pace that most educators are

overwhelmed. S c i e n t i f i c j o u r n a l s alone now number more than 30,000

and 600,000 s c i e n t i f i c papers are being published each year. The

impact upon i n d i v i d u a l s and upon s o c i a l i n s t i t u t i o n s i s only beginning

to be understood.

One of the outcomes of t h i s change has been the r e a l i z a t i o n by

most a d u l t s that they are i n constant need of updated education. "One-

h a l f of what a c o l l e g e graduate l e a r n s i s obsolete i n 10 years; one-

h a l f of what a c o l l e g e graduate needs to know i n 10 years i s not yet

a v a i l a b l e " (3:213). I t i s no longer p o s s i b l e , i n such h i g h l y developed

c o u n t r i e s as ours, f o r a man or a woman to stop l e a r n i n g . Education

must be a c o n t i n u i n g l i f e l o n g process.

1

2

At the same time there i s an exploding world p o p u l a t i o n which

i s e x e r t i n g intense pressure on an already crowded ed u c a t i o n a l system

w i t h i t s l i m i t e d resources, teacher and space shortages. Thus, the

dilemma: on the one hand more people are demanding access to expanding

knowledge through f u r t h e r education; w h i l e on the other hand, there i s

an acknowledged shortage of educators. Education i t s e l f .must change to

meet the needs and requirements of our s o c i e t y .

A proposed s o l u t i o n i s to u t i l i z e new methods of i n s t r u c t i o n

designed to i n c r e a s e the e f f i c i e n c y of communication. " E d u c a t i o n a l

technology, a systematic method of p r o v i d i n g s o l u t i o n s that are tech-

n i c a l l y and economically f e a s i b l e to e d u c a t i o n a l problems, i s the means

by which e d u c a t i o n a l i n n o v a t i o n can most e f f i c i e n t l y and e f f e c t i v e l y

be implemented'.'" (3:214) . Programmed i n s t r u c t i o n i s one of the new

technologies considered by many to be an e f f e c t i v e instrument i n meeting

the demands of contemporary education. "Some e n t h u s i a s t i c advocates

of t h i s method of i n s t r u c t i o n have h a i l e d programmed i n s t r u c t i o n as

the most important e d u c a t i o n a l development s i n c e the i n v e n t i o n of the

p r i n t i n g press" (25:1). At the l e a s t , there i s l i t t l e doubt that "pro-

grammed i n s t r u c t i o n has emerged as the f i r s t t rue system of i n s t r u c t i o n a l

technology education has had" (17:VI) and i t i s capable of making a

s u b s t a n t i a l c o n t r i b u t i o n to the educational process.

"Programmed i n s t r u c t i o n i s the f i r s t a p p l i c a t i o n of l a b o r a t o r y

techniques, u t i l i z e d i n the study of the l e a r n i n g process, to the p r a c t i c a l

problems of education" (:(19YVII). I t can be described as a systematic

methodological approach to the education of the i n d i v i d u a l l e a r n e r ,

3

based upon a psychological analysis of the teaching-learning process.

The major p r i n c i p l e s of learning which underlie the concept of pro-

grammed i n s t r u c t i o n , stated i n general terms, are: (1) planned sequenced

i n s t r u c t i o n based on well-defined behavioural objectives; (2) a c t i v e

response by the learner at each step of the program; (3) immediate feed-

back to the response; (4) self-paced;;;;andd(5)'i)empir.iGaiLly.ydeve!Lopedf1

through a v a l i d a t i o n procedure. Thus, programmed i n s t r u c t i o n with i t s

c o r r e l a t i v e technology o f f e r s education a challenge; i f further research

continues to bear out the p o s i t i v e preliminary findings then new pro-

gramsieoughthtotbebdevel'Op.ededandnduir^eduG

allow for t h e i r presentation. At present t h i s appears to be an obvious

and plausable s o l u t i o n to the numerous education and t r a i n i n g problems

of today.

The Problem

The study problem was to develop a programmed i n s t r u c t i o n

booklet on the topic Methods of Adult Education based, p r i m a r i l y , on

the Verner conceptuallsGheme.9.,anddt0.otes,fetits.steachMggeffecdivenesss

through a v a l i d a t i o n process. The booklet was designed for the use

of u n i v e r s i t y students studying to be future adult educators. Many of

the students are themselves adults, having returned to u n i v e r s i t y to

upgrade t h e i r q u a l i f i c a t i o n s or to educate themselves for a new career.

Importance for Adult Education:

The education explosion taking place today i n adult education

i s without precedent i n the world's h i s t o r y . Adult educators, faced

4

w i t h one of the greatest e d u c a t i o n a l jobs ever given a group of educators,

are beginning to t u r n to programmed i n s t r u c t i o n as a todllinnsoOivihg,

the problem. Business, i n d u s t r y and the m i l i t a r y are already making

extensive use of i t and, f i n a l l y , a d u l t educators are f o l l o w i n g t h e i r

l e a d .

A d u l t educators have long recognized that a d u l t education i s

uniquely d i f f e r e n t from the education of c h i l d r e n . The a d u l t l e a r n e r

b r i n g s w i t h him a background of l i f e experiences a c h i l d does not have.

However, the a d u l t has more at stake when he becomes a student. F a i l u r e

can be very traumatic i n terms of self-esteem, job s e c u r i t y , and/or

s o c i a l esteem.

Programmed i n s t r u c t i o n appears p a r t i c u l a r l y w e l l s u i t e d to the

needs of the a d u l t l e a r n e r . The hetefogeneityoSfaadihltlibearinersccreates

a need f o r i n d i v i d u a l i z e d l e a r n i n g programs designed to meet the s p e c i f i c

goals of each student. In a d d i t i o n , programmed m a t e r i a l s a l l o w the

a d u l t a chance to progress a t h i s own r a t e of speed. Perhaps the most

important b e n e f i t of programmed i n s t r u c t i o n i s that i t has given many

ad u l t s t h e i r f i r s t f e e l i n g s of success i n a l e a r n i n g s i t u a t i o n . Pro-

grammed i n s t r u c t i o n e l i m i n a t e s much of phecpsy.ohologicailikrfskt.fiacformfrom

education and can provide most a d u l t l e a r n e r s w i t h the kind of experiences

needed to ensure success.

Despite these f a c t s , f u t u r e a d u l t educators are not re q u i r e d to

r e c e i v e systematic t r a i n i n g i n the use of programmed i n s t r u c t i o n m a t e r i a l s .

Nor are these m a t e r i a l s i n c l u d e d i n t h e i r r e q uired e d u c a t i o n a l experiences.

F a m i l i a r i t y w i t h azdevice i s necessary before an o b j e c t i v e c o n s i d e r a t i o n

5

of i t s value as a v i a b l e i n s t r u c t i o n a l t o o l can be made. I f programmed

i n s t r u c t i o n i s to be used by these students i n t h e i r future teaching

r o l e , programmed materials should be included i n t h e i r undergraduate

course work. In addition, programmed i n s t r u c t i o n can make a s i g n i f i -

cant co n t r i b u t i o n to the content areas of the student's own learning

program. The problem i s the lack of av a i l a b l e materials. For these

reasons i t was decided to construct and v a l i d a t e a programmed i n s t r u c -

t i o n booklet.

D e f i n i t i o n of Terms

1. Programmed i n s t r u c t i o n i s "any form of pre-prepared, pre-

sequenced i n s t r u c t i o n directed toward a s p e c i f i c educational o b j e c t i v e " :

(25:V), that has the following e s s e n t i a l c h a r a c t e r i s t i c s : (1) written

behavioural objectives, (2) l o g i c a l l y sequenced units of information,

(3) active response by the student (4) immediate feedback to the student,

(5) emperically tested and (6) self-paced. I t i s e s s e n t i a l l y a question

and answer method of i n s t r u c t i o n that can be transmitted i n almost any

sort of f a b r i c t e a c h i n g machine, books, f i l m s , e t c . , a n d can be

auditory, v i s u a l , simulatory and/or d e s c r i p t i v e .

2. BranchedgPr'Ogrammi'ngg, aeco-rddln'ggtooitssinventoKrNN'rmanii

Crowder, 1954, has no learning theory base but i s founded on the con-

cept of perceptive learning: you lea r n that which you perceive. In-

s t r u c t i o n i s oegaQi'zededintotdog

6

of a multiple choice question, and the student i s directed to the next

l o g i c a l i n s t r u c t i o n a l unit for him based on the correctness of h i s

answer.

3. V a l i d a t i o n Process. A process through which a program i s

e m p i r i c a l l y tested on a sample student target population, before

p u b l i c a t i o n , to point out areas of weakness needing r e v i s i o n and to

determine the program's effectiveness as a teaching instrument.

4. C r i t e r i o n Test. A test given to the student upon comple-

t i o n of a program which requires the student to perform the behaviour

i d e n t i f i e d i n the objectives. I t s purpose i s to measure for achievement

of the objectives by the student and to t e s t s t h e f e f l e c t i v e n e s s e o f s s c"

the program.

Plan of the Thesis

The remainder of t h i s thesis i s organized as follows: Chapter

II reviews the research i n three selected areas of programmed i n s t r u c -

t i o n , Chapter III describes i n d e t a i l the procedure used for the con-

s t r u c t i o n and v a l i d a t i o n of programmed i n s t r u c t i o n materials, the data

from the developmental and f i e l d tests are analyzed i n Chapter IV, and

a summary of the r e s u l t s , conclusions and implications for further

research are contained i n Chapter V. The booklet, Methods of Adult

Education coffiprissAAfp"eridi-xAA.

CHAPTER I I

REVIEW OF THE LITERATURE

The f i r s t reported s c i e n t i f i c experiment using programmed

mafcMmeswwasbbyLLitfc&ei&nlit9M

using machines as i n s t r u c t i o n a l aids f o r a d u l t s studying e d u c a t i o n a l

psychology. The c o n c l u s i o n , that programmed m a t e r i a l s f a c i l i t a t e d

l e a r n i n g , demonstrated the p o t e n t i a l programmed i n s t r u c t i o n could have

f o r education. However, p r i o r to 1948 only s i x a r t i c l e s on programs

or programmed i n s t r u c t i o n a l devices appeared i n education l i t e r a t u r e .

In the e n t i r e decade preceding 1958 only 37 s t u d i e s were reported i n

t h i s f i e l d . An e x p l o s i o n of i n t e r e s t i n programmed i n s t r u c t i o n began

i n the m i d - f i f t i e s ; and 37 s t u d i e s are reported i n 1958 (1:16). The

trend has continued unabated to present day, as educators have begun

to r e a l i z e the p o t e n t i a l i t i e s programmed i n s t r u c t i o n has f o r our

education system.

Much of the i n i t i a l research r e l a t i n g to programmed i n s t r u c t i o n

was designed to determine whether or not programmed m a t e r i a l s do, i n

f a c t , teach. There i s no doubt that the multitude of s t u d i e s have

accumulated a convincing array of evidence showing that programmed

m a t e r i a l s are e f f e c t i v e teaching devices. Two a d d i t i o n a l conclusions

that areaapparent from a survey of t h i s research i s the e f f e c t i v e n e s s

of programmed m a t e r i a l s f o r a v a r i e t y of d i f f e r e n t s u b j e c t s , e.g.,

adademic, r e c r e a t i o n a l , v o c a t i o n a l t r a i n i n g , and f o r a v a r i e t y of

7

8

d i f f e r e n t groups of l e a r n e r s , e.g., c h i l d r e n (both normal and exception-

a l ) , c o l l e g e students and a d u l t s (42:103). This review w i l l e s s e n t i a l l y

focus on those s t u d i e s i n programmed i n s t r u c t i o n which r e l a t e s p e c i -

f i c a l l y to a d u l t students i n three areas: ( i ) how does programmed

i n s t r u c t i o n compare w i t h conventional i n s t r u c t i o n a l techniques f o r

teaching e f f e c t i v e n e s s ; (2) how do the two major i n s t r u c t i o n a l paradigms,

l i n e a r and branching, compare and; (3) what has research shown con-

cerning two of the v a r i a b l e s of program c o n s t r u c t i o n ; namely, response

mo.de and s i z e of step?

Programmed vs. Conventional I n s t r u c t i o n

In view of the growing i n t e r e s t i n teaching machines and pro-

grammed i n s t r u c t i o n i t i s important to determine whether they are, as

i s commonly b e l i e v e d , a more e f f e c t i v e or more e f f i c i e n t method of

i n s t r u c t i o n than conventional teacher-dominated classroom techniques.

There has been a m u l t i t u d e of s t u d i e s conducted on t h i s research

question. " I n f a c t , the comparison study has been the most popular

experiment i n the h i s t o r y of programmed i n s t r u c t i o n " (20:167). S e v e r a l

authors, such as Stolurow (52) and H a r t l e y (20) f e e l t h i s i s an inappro-

p r i a t e question and research y i e l d s meaningless data because of the

f a i l u r e to d e f i n e conventional i n s t r u c t i o n , and thus,, i t i s u s u a l l y

t r e a t e d as analogous to l e c t u r i n g . However, an examination of the

vast number and v a r i e t y of experiments on t h i s q u e stion provides a b a s i s

f o r making some conclusions.

A s e r i e s of three experiments were conducted i n October, Novem-

ber and December 1960, by the I n t e r n a t i o n a l Business Machine Corporation

9

(IBM) using 112 IBM and customer engineers. S i x experimental c l a s s e s

t o t a l i n g seventy men were i n s t r u c t e d by means of a programmed textbook

o n l y , w h i l e two c o n t r o l c l a s s e s of forty-two men were i n s t r u c t e d by

the l e c t u r e d i s c u s s i o n technique. The r e s u l t s showed that programmed

i n s t r u c t i o n reduced the t r a i n i n g time needed by 27%. On a comprehensive

.est osthefimaterialccoveredvethe gexperimentatbmgrouposhowedwadleafning

gain of 10% above the c o n t r o l group w i t h a smaller d i s p e r s i o n of scores.

In a d d i t i o n , 83% of the students favoured using programmed i n s t r u c t i o n

i n f u t u r e courses i n place of conventional i n s t r u c t i o n , and 93% found

i t l e s s d i f f i c u l t . The r e s u l t s obtained i n d i c a t e the p o t e n t i a l advantage

i n time, cost and achievement that "programmed i n s t r u c t i o n has f o r the

t r a i n i n g of ad u l t s (26:161-197).

S e v e r a l other researchers, Day (42:13), B l y t h (25:45), Hughes

and MacNamara (34:227), have noted t h i s r e l a t i o n s h i p between programmed

i n s t r u c t i o n and higher l e a r n i n g scores when compared w i t h conventional

i n s t r u c t i o n . In each of the experiments the programmed i n s t r u c t e d

groups showed higher l e a r n i n g achievement scores than the c o n v e n t i o n a l l y

taught groups, and the researchers used t h i s as evidence of a s u p e r i o r

teaching e f f e c t i v e n e s s w i t h programmed i n s t r u c t i o n . The r e s u l t s of a

comparison study c a r r i e d out at Du Pont, using j u n i o r mechanics i n an

engineering course designed to teach the men how to read engineering

drawings, are summarized i n Table I .

Those who re c e i v e d programmed i n s t r u c t i o n spent 25% l e s s time

i n l e a r n i n g the subj e c t and achieved s i g n i f i c a n t l y higher scores on

the p o s t - t e s t than those who re c e i v e d conventional group i n s t r u c t i o n (34).

10

Table I

Comparison of Programmed I n s t r u c t i o n and Conventional I n s t r u c t i o n *

Programmed Conventional Program vs. Method Method Convention

Average man hours per t r a i n e e 12.8 17.0 25% l e s s

Average examination scores 91% 81% 13% more

However, not a l l researchers agree w i t h the above conclusions.

I n experiments conducted by Wendt, Hough, Larue and Donelson, and Smith

(45, 23, 34, 4(E) there was no increase i n l e a r n i n g noted f o r the pro-

grammed i n s t r u c t i o n groups, although a l l reported evidence s t r o n g l y

i n d i c a t e d that the time r e q u i r e d to achieve t h i s l e a r n i n g can be r e -

duced through the a p p l i c a t i o n of programmed i n s t r u c t i o n . In a compara-

t i v e experiment by Maurice Larue and E l a i n e Donelson designed to

determine the e f f i c i e n c y of a programmed textbook, a teaching machine,

and conventional classroom l e c t u r e 70 a d u l t male m i l i t a r y personnel

completed a nineteen hour segment of a Fundamentals of Computers s e c t i o n

i n a weapon system t r a i n i n g course. There was no s i g n i f i c a n t d i f f e r e n c e

i n gain achievement scores between the three groups. However, the

machine t r a i n e d group showed a 24% time saving and the programmed book

group 28% when compared to the l e c t u r e group, without any r e d u c t i o n i n

p r o f i c i e n c y (34:241-246). No s i g n i f i c a n t d i f f e r e n c e i n l e a r n i n g achieve-

ment or r e t e n t i o n between groups i n s t r u c t e d i n conventional classroom

l e c t u r e s when compared to those i n s t r u c t e d by programmed textbooks were

11

noted i n experiments using a i r f o r c e p o l i c e (25:52), i n s t a l l e r - r e p a i r -

men at General Telephone (25:45), A i r Force cadets (41), and u n i v e r s i t y

freshmen (45) However, a l l experiments noted the t r a i n i n g time

re q u i r e d to o b t a i n the same l e v e l of e f f i c i e n c y was s u b s t a n t i a l l y

reduced i n the programmed i n s t r u c t i o n groups; the a i r force p o l i c e

t r a i n i n g by 1/2 to 2/3, the i n s t a l l e r repairmen by 40-45%, the a i r

fo r c e cadets by 44-47% and the u n i v e r s i t y freshmen r e q u i r e d 1/4 the

time taken by t h e i r c o n v e n t i o n a l l y taught counterparts. Thus programmed

i n s t r u c t i o n techniques were found to be conside r a b l y more e f f i c i e n t

than the conventional l e c t u r e - d i s c u s s i o n i n s t r u c t i o n a l techniques, when

e f f i c i e n c y was defined as a saving of i n s t r u c t i o n a l time.

Bushness made an i n t e r e s t i n g f i n d i n g i n a comparative e x p e r i -

ment using a group of a d u l t workers experienced i n c o n t i n u i n g education.

N i n e t y - s i x journeymen e l e c t r i c i a n s were assigned to one of three modes

of i n s t r u c t i o n f o r three hours a week f o r eighteen weeks. The three

modes of i n s t r u c t i o n i n c l u d e d , (1) i n d i v i d u a l i z e d s e l f - p a c e d programmed

i n s t r u c t i o n , (2) combination of programmed i n s t r u c t i o n and i n s t r u c t o r

lead d i s c u s s i o n and, (3) conventional l e c t u r e - d i s c u s s i o n technique w i t h

the use of advanced a u d i o v i s u a l a i d s . A s i g n i f i c a n t saving of i n s t r u c -

t i o n a l time favouring the two groups using programmed i n s t r u c t i o n was

found, but i n the combination group that p a i r e d programmed i n s t r u c t i o n

w i t h d i s c u s s i o n , the t r a i n i n g time was reduced the most, w h i l e the

students were more h i g h l y motivated and more p o s i t i v e (34:110-113).

These s t u d i e s were chosen to i l l u s t r a t e the wide range of

a p p l i c a t i o n s programmed i n s t r u c t i o n has f o r a d u l t education. However,

12

there are a number of l i m i t a t i o n s to these comparison experiments.

They are d i f f i c u l t to c a r r y out p r e c i s e l y because of the number of

v a r i a b l e s i n t e r a c t i n g i n two groups that can never be f u l l y i t e m i z e d

or c o n t r o l l e d , e.g., the n o v e l t y of u s i n g programmed i n s t r u c t i o n

m a t e r i a l s has an unknown i n f l u e n c e on students. I n s i n g l e comparison

s t u d i e s n e i t h e r the method used nor the teacher i n v o l v e d can be repre-

s e n t a t i v e of a l l methods oat a l l teachers. Therefore, the r e s u l t s of

any s i n g l e comparison study must remain s p e c i f i c to that study as

c a r r i e d out, and g e n e r a l i z a t i o n s must be made w i t h c a u t i o n .

Despite the f a c t that i t i s not i n p r i n c i p l e d i f f i c u l t to do

a comparison experiment, i t i s d i f f i c u l t to do i t p r e c i s e l y , and c l e a r l y ,

i t i s r i s k y to g e n e r a l i z e from the r e s u l t s of a s i n g l e study. A d i f f e r -

ent approach would be to look at a l a r g e number of these comparison

experiments and consider g e n e r a l i z a t i o n s that emerge. H a r t l e y (20)

reviewed 112 such s t u d i e s and the r e s u l t s are shown i n Table 2.

Table 2

The R e s u l t s of 112 Studies Comparing Programmed I n s t r u c t i o n w i t h Conventional I n s t r u c t i o n

Programmed I n s t r u c t i o n Group Number of Studies No

Measures Recording S i g n i f i c a n t l y S i g n i f i c a n t S i g n i f i c a n t l y Recorded These Measures Superior D i f f e r e n c e Worse Time taken 90 Test r e s u l t s 110 Retest r e s u l t s 33

Note: Figures i n the 1st column d i f f e r because not a l l three measures are recorded f o r everyone of the 112 s t u d i e s .

47 37 6 41 54 15 6 24 .3

13

These s t u d i e s , i n g e n e r a l , tend to support the f i n d i n g s of the

other s t u d i e s described i n t h i s s e c t i o n , and the f o l l o w i n g summary

statements may be drawn from the data:

1. Programmed i n s t r u c t i o n i s e q u a l l y as e f f e c t i v e an i n s t r u c -

t i o n a l technique as conventional classroom techniques, and i n many

instances has been found to be s i g n i f i c a n t l y more e f f e c t i v e .

2. Programmed i n s t r u c t i o n , i n most cases, has r e s u l t e d i n a

saving of i n s t r u c t i o n a l time r e q u i r e d to achieve the same l e v e l of

p r o f i c i e n c y . The major i m p l i c a t i o n of these s t u d i e s i s that programmed

i n s t r u c t i o n worked as w e l l as, or b e t t e r t h a n , c c o n v e n t i o n a l i i n s t r u c t i o n

i n a d u l t education programs.

Branching Vs. L i n e a r Paradigms

Two b a s i c auto i n s t r u c t i o n a l models have been i d e n t i f i e d , (1)

the l i n e a r model which u s u a l l y d i s p l a y s a f i x e d sequence of l e a r n i n g

s t e p s , r e q u i r i n g a l l subjects to t r a v e r s e a l l items, and (2) Crowder's

branching model which uses a v a r i a b l e sequencing of items, based on

student response to posed m u l t i p l e choice questions. A review and

assessment of the research comparing the two modes of program presenta-

t i o n was conducted.

The l i t e r a t u r e search revealed that most of these experiments

were c a r r i e d out over a comparatively narrow f i e l d . Many i n v e s t i g a t o r s

used only s m a l l numbers of students, others d e a l t w i t h only one type

of l e a r n e r , or a very l i m i t e d number of programs, or only one type of

s u b j e c t matter. In almost a l l cases only one aspect was examined and

14

the p o s s i b l e I n f l u e n c e of the other v a r i a b l e s was ignored. I t i s not

s u r p r i s i n g , t h e r e f o r e , that the r e s u l t s are o f t e n i n c o n c l u s i v e and the

f i n d i n g s of one experimenter are oft e n r e f l e c t e d by another i n the

same f i e l d (22:83). However, i t i s worthwhile to examine some of the

b e t t e r s t u d i e s i n order to note any g e n e r a l i z a t i o n s which may emerge.

Coulson and Silberman ( 8 ) , using students e n r o l l e d i n a psychology

course at Harvard U n i v e r s i t y , i n v e s t i g a t e d three dimensions along which

one teaching machine may d i f f e r from another, (1) student response mode,

(2) s i z e of item step and (3) item sequence c o n t r o l , or f i x e d sequence

vs. branching. The ei g h t y subjects were t r a i n e d under one of e i g h t

d i f f e r e n t t r a i n i n g procedures, the e i g h t t r a i n i n g procedures represent-

in g the e i g h t combinations of the three experimental v a r i a b l e s . The

r e s u l t s f o r v a r i a b l e s (1) and (2) w i l l be discussed l a t e r . The branch-

i n g f a c t o r d i d not appear to a f f e c t l e a r n i n g achievement Scores when

compared to the f i x e d sequence, but made p o s s i b l e a s i g n i f i c a n t decrease

i n r e q u ired t r a i n i n g time. The researchers concluded that when both

the amount learned and the r e q u i r e d t r a i n i n g time are considered, the

branching procedure appears to o f f e r an o v e r a l l advantage over non-

branching. Silberman et a l . , (39), i n two experiments that were con-

cerned w i t h methods of branching, i n d i c a t e d that there i s a r e l a t i o n s h i p

between f l e x i b i l i t y of m a t e r i a l s and the r a t e of l e a r n i n g that favours

the f l e x i b l e m a t e r i a l s over the f i x e d sequence. Backward branching,

however, which allows review o n l y , was not found to be s u p e r i o r to the

f i x e d sequence. These conclusions are r e f l e c t e d by H a r t l e y (21), who

compared the e f f e c t i v e n e s s of a l i n e a r ppogram e n t i t l e d , "A R e v i s i o n

15

Course i n Logarithms" w i t h that of an o p t i o n a l branch program on the

same sub j e c t . The r e s u l t s showed both programs were e q u a l l y e f f e c t i v e ,

and there was no s i g n i f i c a n t d i f f e r e n c e between the times taken to

achieve the r e s u l t s .

One hundred and twenty students from Ohio U n i v e r s i t y , r e g i s t e r e d

i n I n d u s t r i a l A r t s and I n d u s t r i a l Technology courses, were subjects i n

an experimental study by S k u l l (40) to determine which of two programs,

l i n e a r or branching, would be more e f f e c t i v e i n the teaching and r e t e n -

t i o n of s e l e c t e d t e c h n i c a l i n f o r m a t i o n to high and low a c h i e v i n g c o l l e g e

students. S k u l l found a s i g n i f i c a n t d i f f e r e n c e i n l e a r n i n g occurred,

i n favour of the branching group, although t h i s d i f f e r e n c e d i d not appear

on the r e t e n t i o n t e s t s .

F o l l o w i n g a l i t e r a t u r e review of s t u d i e s which i n v e s t i g a t e d

t h i s question Arnold Rae (29) concluded that past s t u d i e s revealed no

s i g n i f i c a n t d i f f e r e n c e i n the t e r m i n a l performances of students who

worked through l i n e a r or branching programs, but branching programs

req u i r e d l e s s time to complete the l e a r n i n g s e s s i o n . However, an

experiment conducted by Rae using 189 freshmen engineering students

from U.C.L.A. showed no d i f f e r e n c e i n e i t h e r l e a r n i n g time or t e s t scores

between the branching or the l i n e a r methods.

The e s s e n t i a l elements of a number of s t u d i e s have been examined

i n which the r e l a t i v e e f f e c t i v e n e s s of l i n e a r and branching programs

have been compared. An u l t i m a t e c r i t e r i a f o r determining the r e l a t i v e

merits of programs i s mastery of the subject matter as evidenced by

f i n a l c r i t e r i o n t e s t scores. The evidence i s confusing and c o n t r a d i c t o r y .

16

At the present time, therefore, i t can be said there i s no experimental

evidence that c l e a r l y supports a claim to superiority by either li n e a r

or branching programs (25:14). However, perhaps there are certain

circumstances i n which one type of program would be more effec t i v e than

the other.

J. Hebenton (22) was able to organize a programmed learning

scheme which was both broadly based and of comparative long duration

(over 30,000 student study hours). The students ranged from the semi-

s k i l l e d workers to clerks and typists to engineering graduates.and

executives. The subject matter had an equally wide front, and varied

from academic subjects, (>egg,,hb-asicEEnglis'lj) ftoraiM$3geffineu&dBj:efes ^egg.^nnetwor.kaanalysVis) ^otdeleisureus'ubj:e"ctse(g.,g. sagaiMng) Jhexfieepre-

sentation modes included books and machines, l i n e a r and branching.

Because the obijeettwassto obtain results from the natural programmed

learning s i t u a t i o n , nothing was suggested to the student that an experi-

ment was i n progress. Relevant factors i n the choice of l i n e a r vs.

branching were found and grouped under three headings as follows:

student (1) student: i n t e l l i g e n c e l e v e l , educational standard, verbal

a b i l i t y .

(2) s-ubject matter: whether concrete or abstract, p r a c t i c a l

or theoretical.

(3) purpose: memorizing, understanding, forming judgements.

The results produced a guide to the optimum use of l i n e a r vs. branching

programs and are summarized below:

17

L i n e a r

Low

Low

Low

Concrete

P r a c t i c a l

Memorizing

Information

Parameters Relevant to L i n e a r Vs. Branching

I n t r i n s i c or Branching

student i n t e l l i g e n c e

student emotional l e v e l

student v e r b a l l e v e l

s ubject

subject

purpose

purpose

hig h

high

h i g h

a b s t r a c t

t h e o r e t i c a l

unders tanding

judgement

Thus, research f a i l s to support a c o n c l u s i o n that e i t h e r l i n e a r

or branching programs are i n h e r e n t l y b e t t e r than the other but parameters

have been e s t a b l i s h e d which serve as a u s e f u l guide f o r program w r i t e r s

and program users.

V a r i a b l e s i n Program Co n s t r u c t i o n

R e l i a b l e c r i t e r i o n measures, which define a good program frcx

shoM'dfSecbasedoon,researchbfihdingson Thee-inffliuenceiof.stheTmany.nfTc

M d ^ e n i e f l } vaciabiLesntforrexample ,erespom'sepmode,mpacingacfegd-f e e

18

t h a t might emerge. This review w i l l focus on those s t u d i e s i n pro-

grammed i n s t r u c t i o n that r e l a t e s p e c i f i c a l l y to two v a r i a b l e s : response

mode and step s i z e .

Overt an Covert Responses Overt vs. Covert Responses

The i s s u e of response mode i s fundamental to general l e a r n i n g

theory. The importance and f u n c t i o n of overt responses i n l e a r n i n g

has been debated by both l e a r n i n g theoristsaarideeducators,aaridiisaa

b a s i c and p e r s i s t e n t disagreement among l e a r n i n g t h e o r i s t s . " I t separates

those who i d e n t i f y w i t h the S-S theory from those which i d e n t i f y w i t h

an S-R theory" (44:421). I t would be u s e f u l , t h e r e f o r e , to determine

the degree to which f i n d i n g s on mode of response,oorccovert vs. overt

responses, apply to programmed i n s t r u c t i o n . This v a r i a b l e i s of par-

t i c u l a r i n t e r e s t s i n c e i t forms a p o i n t of departure between l i n e a r and

branching programs, l i n e a r g e n e r a l l y u s i n g a constructed response and

branching a m u l t i p l e choice technique.

A great number of stu d i e s have been designed to y i e l d data

regarding the comparative e f f e c t i v e n e s s of constructed and s e l e c t e d modes

of response i n programmed i n s t r u c t i o n m a t e r i a l s . In a study u s i n g

u n i v e r s i t y students studying the h i s t o r i c a l foundations of the modern

secondary school c u r r i c u l u m Hough (23) reported no d i f f e r e n c e s on the

c r i t e r i o n t e s t between students taught by means of constructed and

s e l e c t e d response programs. Fry (16) on the other hand, found that i n

the teaching of Spanish vocabulary to high school students the constructed

response mode was more e f f e c t i v e when the c r i t e r i o n of l e a r n i n g was

19

r e c a l l , but there was no d i f f e r e n c e when the c r i t e r i o n t e s t was of the

m u l t i p l e choice v a r i e t y . The covert response mode was found to take

s i g n i f i c a n t l y l e s s time f o r completion of programmed m a t e r i a l s without

corresponding decrease on theeimmedii!a,teerefeent6.nnmeas.ur.eeinnann

experiment by Lambert, M i l l e r and Wiley. (3$i)i . anlnxanrexperimentaalready

d e s c r i b e d , by Coulson and Silberman (8) which i n v e s t i g a t e d three

v a r i a b l e s along which one teaching machine may d i f f e r from another, the

response mode d i d not s i g n i f i c a n t l y a f f e c t the amount le a r n e d , although

the r e q u i r e d t r a i n i n g time was considerably l e s s f o r m u l t i p l e choice

responses over constructed responses. Thus, the o v e r a l l advantage

appears to be w i t h the m u l t i p l e choice mode. S i m i l a r f i n d i n g s are

reported by Rae (25:46) using engineering students at U.C.L.A. as

s u b j e c t s , S t o l u r o n and Walder (44) using content from a d e s c r i p t i v e

s t a t i s t i c a l course, and Paschal Strong (34) f o l l o w i n g a review of nine

r e c e n t l y p ublished s t u d i e d i n v o l v i n g m u l t i p l e choice vs. w r i t t e n response.

Evans, Glaser and Homme (7:543) conducted a study at the

U n i v e r s i t y of P i t t s b u r g h u s i n g content c a l l e d 'Fundamentals of Music'

to compare the e f f e c t i v e n e s s of the c o n s t r u c t i o n method of responding

w i t h no overt response. The group making no overt response spent l e s s

time i n l e a r n i n g but the response mode d i d not s i g n i f i c a n t l y i n f l u e n c e

f i n a l t e s t scores. A f i n d i n g c o n s i s t e n t w i t h these r e s u l t s was obtained

by the same researchers (14) when a program designed to teach the con-

s t r u c t i o n of short deductive proofs i n symbolic l o g i c was administered

to c o l l e g e students. I t appeared that w i t h i m p l i c i t responses subjects

completed t h e i r programs i n about 65% of the time taken by students who

20

had to record t h e i r responses.

These s t u d i e s , comparing overt and covert responses i n d i c a t e

that students proceed through a program more r a p i d l y w i t h covert

responses. Comparison of the post t e s t performance of students employ-

i n g no w r i t t e n responses w i t h that of students who learned by making

w r i t t e n responses have g e n e r a l l y y i e l d e d e q u i v o c a l r e s u l t s . Eigan and

Margulus (34) attempted to account f o r these divergent r e s u l t s by

va r y i n g two response c h a r a c t e r i s t i c s , the relevance of the re q u i r e d

response to the m a t e r i a l taught and the i n f o r m a t i o n l e v e l of the r e -

quired response. While no s i g n i f i c a n t d i f f e r e n c e s were observed between

response modes at the low in f o r m a t i o n l e v e l so at the intermediate or

high i n f o r m a t i o n l e v e l s when the responses were i n c i d e n t a l , overt

responses y i e l d e d more s u p e r i o r student performance f o r intermediate

and high i n f o r m a t i o n l e v e l s when the responses were r e l e v a n t . Thus i t

appears the best response mode depends on p a r t i c u l a r s i n the s p e c i f i c

l e a r n i n g s i t u a t i o n , e s p e c i a l l y the amount of time a v a i l a b l e and d i f f i -

c u l t y of the content.

The experimental evidence comparing covert and overt modes of

response i s i n c o n s i s t e n t . Consequently, absolute conclusions are not

p o s s i b l e . However, i n general, i t would appear c o v e r t t r e s p o n s e s s o f f e r a

savings i n time when compared to overt responses, without any l o s s i n

leaEndingppcof'engy. As a consequence, when time i s taken i n t o account,

covert responses tend to be more e f f i c i e n t inttermsoofaamouhtllearned

per u n i t then response modes r e q u i r i n g overt behavior (37, 17). This

i s not an i n d i s c r i m i n a t e f i n d i n g , and ' i t i s necessary to consider

21

p a r t i c u l a r s of the s p e c i f i c l e a r n i n g s i t u a t i o n , namely, the i n t e l l i g e n c e

l e v e l of the l e a r n e r , the d i f f i c u l t y of the l e a r n i n g t a s k , and the

amount of time a v a i l a b l e .

However, these f i n d i n g s do have important i m p l i c a t i o n s f o r pro-

grammed i n s t r u c t i o n . I t appears that overt responding to every frame

i s not e s s e n t i a l f o r e f f e c t i v e l e a r n i n g (25:67). Students motivated

to l e a r n can a c t i v e l y respond i n ways other than a c t u a l l y w r i t i n g

answers, such as t h i n k i n g . This can not only r e s u l t i n a savings i n

l e a r n i n g time, but a savings i n teaching m a t e r i a l s . The conservation on

cost of programmed textbooks by simply r e q u i r i n g covert responses of

the students, thus a l l o w i n g the reuse of programmed textbooks, should

be an important c o n s i d e r a t i o n f o r programmers.

Large Step Vs. S a a l l Step Large Step vs. Small Step

One of the e s s e n t i a l c h a r a c t e r i s t i c s of programmed m a t e r i a l s

i s t hat a r e l a t i v e l y s m a l l u n i t of sequenced i n f o r m a t i o n i s presented

to the student at a time, tQowhichhth'e esltudent must respond before

being presented w i t h the second u n i t of i n f o r m a t i o n . I t i s common

p r a c t i c e to l i m i t each p r e s e n t a t i o n frame to cover only one concept

or p a r t of a concept. A b a s i c question to be asked i s : what g u i d e l i n e s

does research o f f e r concerning the s i z e of step? Are l a r g e steps more

or l e s s e f f e c t i v e than s m a l l ones?

There has been a m u l t i t u d e of s t u d i e s r e p o r t i n g on t h i s question.

In one of the f i r s t experiments designed to provide a f u n c t i o n a l a n a l y s i s

of the process of programming, Evans, Glaser and Homme (7), 1959,

22

i n v e s t i g a t e d the e f f e c t of the number of steps i n a program upon l e a r n -

i n g time, frequency of e r r o r s during l e a r n i n g and upon immediate and

delayed t e s t performance. The r e s u l t s show t h a t , w i t h i n l i m i t s ,

i n c r e a s i n g the number of steps i n a program r e s u l t e d i n decreases i n

the number of e r r o r s on immediate and delayed performance t e s t s . I n

a d d i t i o n , s m a l l e r steps, e.g., the use of more items to cover the same

subject matter, r e s u l t e d i n l e s s time per step, but greater t o t a l

l e a r n i n g time, and fewer e r r o r s occurred during the course of l e a r n i n g .

The authors p o i n t out that the optimum s i z e of step might be expected

to vary as a f u n c t i o n of the type of subject matter being programmed.

Beyond a c e r t a i n p o i n t i n c r e a s i n g the number of steps d i d not r e s u l t

i n improved performance.

One of the few pieces of s o l i d experimental evidence t r e a t i n g

v a r i a t i o n s i n length of step a n a l y t i c a l l y comes from the experiments

of Maccaly, S h e f f i e l d , and Margolius (29:533) w i t h step by step pro-

cedural demonstrations. The experimenters found that short steps

were b e t t e r than long ones, at l e a s t i n i t i a l l y . However, they a l s o

concluded that an optimum p a t t e r n f o r average students would be

enforced progression from s h o r t e r to longer steps. Other experiments

undertaken to date agree w i t h t h i s c o n c l u s i o n ; that g e n e r a l l y s m a l l

s e q u e n t i a l steps r e s u l t i n more l e a r n i n g than do long steps (30, 13, 20

25, 34). A f u r t h e r f i n d i n g i s t h a t , i n general, s m a l l e r steps r e q u i r e

l e s s time per step but more t o t a l time (30, 13, 20, 25).

In a p r e v i o u s l y described experiment conducted by Coulson and

Silberman ( 8 ) , which i n v e s t i g a t e d three dimensions along which one

23

teaching machine may d i f f e r from another, the s i z e of item step (small

vs. large) was s t u d i e d . The researchers showed the s m a l l step groups

learned more, and concluded that there was a s u p e r i o r i t y i n t o t a l amount

learned w i t h s m a l l steps. This data tends to support one of Skinner's

notions regarding teaching methods, namely the importance of s m a l l steps

i n w r i t i n g items. However, the data a l s o i n d i c a t e s that s i z e of step

may not be as appropriate a d e s c r i p t i o n of what i s i n v o l v e d as redundancy

would be. I t appears that the s m a l l steps are r e a l l y a d d i t i o n a l items

r e l a t i n g to a p a r t i c u l a r concept (42:143).

I t seems apparent that r e s o l u t i o n of the question of s i z e of

step a c t u a l l y i n v o l v e s a complex of f a c t o r s which need to be f u r t h e r

analyzed, and i t can be concluded that more experimentation w i t h b e t t e r

f o r m u l a t i o n of f a c t o r s i s needed before p r e c i s e statements about the

advantage of s m a l l steps over l a r g e steps can be made.

Summary

This b r i e f review of s e l e c t e d s t u d i e s i n v e s t i g a t e d three areas

of research i n programmed i n s t r u c t i o n . In general i t would appear pro-

grammed i n s t r u c t i o n o f f e r s a savings i n time over conventional i n s t r u c t i o n

w i t h no l o s s i n p r o f i c i e n c y , and that n e i t h e r l i n e a r or branching programs

are i n h e r e n t l y b e t t e r than the other. I n a d d i t i o n , covert responses

appear as e f f e c t i v e as overt responses and s m a l l steps are s u p e r i o r to

l a r g e steps. Among the i m p l i c a t i o n s these s t u d i e s have f o r the area of

program technology was the demonstration t h a t a s c i e n t i f i c key to program

c o n s t r u c t i o n has not yet been developed. O b j e c t i v e measures, based on

r e l i a b l e experimental evidence would be of great a s s i s t a n c e i n preparing

programmed m a t e r i a l s and would r e s u l t i n the development of more accurate,

more appropriate and more e f f e c t i v e programs.

CHAPTER I I I

PROCEDURE

Con s t r u c t i o n of the Booklet

The development of programmed i n s t r u c t i o n a l m a t e r i a l s can be

d i v i d e d i n t o four phases: program s e l e c t i o n , program c o n s t r u c t i o n ,

program e v a l u a t i o n and program implementation. Before c o n s t r u c t i o n

of a program on any subject or segment of a course of study begins

there are three b a s i c questions that must be answered: (1) "Should

the subject be programmed, (2) i f the answer i s yes, then what program-

ming technique or paradigm should be used, and (3) what medium or

combination of media should be used to present the programmed mater-

i a l " (12:11). Each of these questions were considered before beginning

c o n s t r u c t i o n of Methods of Adu l t Education, and are discussed here.

Subject S e l e c t i o n

The concept of methods i s course content f o r Education 412

o f f e r e d at the U n i v e r s i t y of B r i t i s h Columbia through Correspondence

Study, the extension d i v i s i o n n i g h t time course, and a r e g u l a r on campus

day course. I t normally r e q u i r e s at l e a s t one c l a s s s e s s i o n to present

the concept of methods and the c l a s s i f i c a t i o n scheme. Because there

are a number of new d e f i n i t i o n s and a cons i d e r a b l e amount of content

a c q u i s i t i o n presented i n a l i m i t e d amount of c l a s s - t i m e , students are

24

25

o f t e n confused and unclear of f a c t s . Because t h i s concept I s fundamental

to the study and understanding of the o r g a n i z a t i o n of a d u l t education

a c t i v i t i e s , students who do not have an exact comprehension of the

d e f i n i t i o n of methods would l i k e l y experience d i f f i c u l t y when reviewing

current l i t e r a t u r e i n the f i e l d of a d u l t education, w r i t i n g course

r e q u i r e d papers, p a r t i c i p a t i n g i n c l a s s d i s c u s s i o n , or p a r t i c i p a t i n g i n

Br-gr-ammpjanningvig.

The d e c i s i o n to programmtheht-opi)ci"MebhoHsd6foAdMelEducatJion'1

was made a f t e r a f e a s i b i l i t y study that showed the c o n s t r u c t i o n of a pro-

gramnwouid'ls^tisf.yfa' p r e s ' e n t n s t u d e r i e n l e a ^

programmed m a t e r i a l s already a v a i l a b l e that could s a t i s f y the i n s t r u c -

t i o n a l requirements. To determine the s u i t a b i l i t y of the t o p i c f o r

programming; that i s , whether the m a t e r i a l to be taught and the teaching

s i t u a t i o n lend themselves economically and e d u c a t i o n a l l y to the develop-

ment of a program, a l i t e r a t u r e review was conducted and a c h e c k l i s t of

suggested c r i t e r i a devised, as f o l l o w s : (see f o l l o w i n g page).

The c r i t e r i o n c h e c k l i s t s t r u c t u r e s the in f o r m a t i o n i n a s y s t e -

matic, convenient and c o n s i s t e n t manner and i s a u s e f u l guide when

determining i f a subject would be appropriate to program. Based on

the r e s u l t s of the c r i t e r i o n c h e c k l i s t i t was decided to program Methods

of Adult Education.

Paradigm S e l e c t i o n

The f i r s t step i n program c o n s t r u c t i o n was to c o n s u l t w i t h

c u r r i c u l u m s p e c i a l i s t s , who helped i d e n t i f y the sources of content on

26

C r i t e r i a f o r S e l e c t i n g Subjects f o r Programming 1

C r i t e r i a Present (yes) Absent (no)

1. I s the subject content r e l a t i v e l y s t a b l e ? /

2. Is the s u b j e c t r e q u i r e d course content? /

3. Are mastery c r i t e r i a f o r the subject r e a d i l y a v a i l a b l e ? /

4. Can the subject be taught w i t h -out the need f o r complementing i n s t r u c t i o n , or t r a i n i n g a i d s , such as awkward or dangerous equipment? /

5. Does the student's education and experience l e v e l permit the use of programmed m a t e r i a l s ? /

6. I s the s u b j e c t taught f r e q u e n t l y enough to j u s t i f y the expense and the development of a program? /

7. I s the student p o p u l a t i o n ade-quate to j u s t i f y the expense? /

8. W i l l the average completion time be l e s s than 10 hours? /

9. I s the subject matter adaptable to programming? /

10. Is there a l e a r n i n g need f o r a program? /

* Sources: 15, 5, 12.

27

the subject to be taught. Once the l i t e r a t u r e review was completed

and a l l r e l e v a n t i n f o r m a t i o n c o l l e c t e d , a general statement of objec-

t i v e s and programmspeGi-f-icat&omsnwa-sadevis^

of the t a r g e t l e a r n e r ' s a b i l i t i e s and t h e i r l e a r n i n g needs on the

sub j e c t . The s e l e c t i o n of programming paradigm or model was based, i n

p a r t , on the i n f o r m a t i o n . "The programming paradigm or model s u p p l i e s

the b a s i c c o n c e p t i c a l framework through which the i n d i v i d u a l items are

connected" (30:40). I t i s chosen w i t h c o n s i d e r a t i o n to the o b j e c t i v e s ,

subject matter, t a r g e t p o p u l a t i o n and a b i l i t i e s of the program w r i t e r .

The v a r i o u s models or paradigms were examined to s e l e c t the

most s u i t a b l e f o r Methods of Adult Education. There are a great v a r i e t y

of programming techniques, but most programs tend to be of two or three

main types. At present "there i s l i t t l e e m p i r i c a l b a s i s to favour one

general type of program over another" (27:28). While research has shown

no s i g n i f i c a n t d i f f e r e n c e i n l e a r n i n g e f f i c i e n c y between l i n e a r and

branching programs (35:73, 37:15, 22:83), the f a c t remains that e i g h t

out of every ten courses produced are l i n e a r (27:73). There i s good

reason to suppose the greater number of l i n e a r programs i s due to the

f a c t that l i n e a r programs are b e l i e v e d by many to be e a s i e r to organize,

use simplerraidssanddareetheecheapestttooreproduGee. Someeauthbrss

conclude that the choice of model i s a matter of personal preference

(35:73) a f t e r c o n s i d e r a t i o n to the various c r i t e r i a l i s t e d p r e v i o u s l y .

Although l i n e a r p r o gramsearerusuaMylconsideredeeffectivevfof

a l l age and a b i l i t y l e v e l s , branching programs are more s u i t e d to the

o l d e r , more i n t e l l i g e n t student (Crowder recommends over age 13), (4:70,

28

22:88) who g e n e r a l l y consider i t more c h a l l e n g i n g and more i n t e r e s t i n g

(12:62). This may be, i n p a r t , because the branching sequence achieves

i t s ends more q u i c k l y than the l i n e a r . In a d d i t i o n , "branching programs

are based on the assumption that t r a i n i n g i s a process of understanding,

where students are f u l l y conscious of the reasons f o r t h e i r c o r r e c t

responses as w e l l as the reasons f o r any e r r o r s . Thus, l e a r n i n g r e s u l t s

from an i n t e r n a l process of reasoning" (37:6). Perhaps t h i s has more

appeal.to the older student.

The subject matter and o b j e c t i v e s place r e s t r a i n t on the choice

of program model. Although the branching model would seem to be p e r t i n e n t

to a wide range of programmed u n i t s , i t has p a r t i c u l a r appeal f o r pro-

grams.einianaareaewherereonstantndecisiohomakihgnis iEeq'u$cede

29

or no o b j e c t i v e date a v a i l a b l e to support these suggestions, and they

must be regarded as t h e o r e t i c a l hypothesis based only on experience

(37:17).

Branching t e x t s have been prepared i n such d i v e r s e f i e l d s as

a l g e r b r a , trigonometry, physics , c b h e M s t r y ,ll'aw,aaridCGont'actbbidldge

(9:296). The branching model was s e l e c t e d because i t appeared to be

the most u s e f u l methodology when c o n s i d e r a t i o n was made of the various

c r i t e r i a . However, i t seems c l e a r that standards f o r the adequacy of a

program ought to be concerned p r i m a r i l y i n terms of i t s e f f e c t i v e n e s s

i n a t t a i n i n g defined e d u c a t i o n a l o b j e c t i v e s r a t h e r than by s p e c i f y i n g

the format, sequencing or other aspect of the means whereby these ends

are achieved (29:566).

The l a s t question to be considered was what medium should be

used to present the program. The m a j o r i t y of programs i n use today are

of the "paper and p e n c i l " book type. This i s , by f a r , the most economical

medium and, because i t r e q u i r e s only the booklet f o r equipment, the

e a s i e s t and most convenient f o r students to use. I t has not been recom-

mended that a beginning programmer, as the w r i t e r , attempt a program

using advanced media (12:16). Since a simple branching booklet does

not r e q u i r e the very s p e c i a l i z e d and lengthy t r a i n i n g necessary to

develop a m a t h e t i c a l or machine a s s i s t e d program, the branching booklet

was s e l e c t e d as more s u i t a b l e to the a b i l i t i e s of the program w r i t e r

and more useable by the students.

30

Content Development

Each step i n programming process leads n a t u r a l l y i n t o the next.

The i n t e r r e l a t i o n s h i p s have been i l l u s t r a t e d %ya]jysaught X>u3i2'4j)iuahd are

represented d i a g r a m a t i c a l l y i n Figure I .

subject s e l e c t i o n

d e f i n i t i o n s and assumptions ( l i t e r a t u r e review)

r e v i s x o n

o b j e c t i v e s

paradigm s e l e c t i o n

e v a l u a t i o n

J i n i t i a l t e s t i n g

o r d e r i n g of subject matter

c o n s t r u c t i o n of items

Figure I . I n t e r r e l a t i o n s h i p s of the Steps i n Programming*

* Source: 30:

31

However, the process i s not merely one d i r e c t i o n a l as suggested

here, and at many points i n the development of Methods of Adult Education

the w r i t e r was forced to reexamine previous steps and make r e v i s i o n s . A

more r e a l i s t i c programming process i s represented diagramatically i n

Figure II as developed by Hartley.(20:15).

objectives target population

development task and behavioural analysis of measures of . prof aicmency

course construction j , 1.

v a l i d a t i o n empirical t e s t i n g and evaluation

t + J

implementation

r e v i s i o n

Figure I I . Programming Process*

* Scarce; 2C

The objectives or aims of the program were s p e c i f i e d p r e c i s e l y

i n measurable behavioural terms as suggested by Rob;at Mager: Preparing

Ins|ructi:o.5a^bQbji'eGtlves. ThTheeterminalrdriteBionefeesfrawasoooris't'r.wet!ed

based on the objectives, to ensure that the knowledge and s k i l l s which

the student i s expected to demonstrate upon mastery of the program would

be tested. Once the s p e c i f i c responses that constitute successful com-

p l e t i o n of the c r i t e r i o n behaviour by the student were i d e n t i f i e d a

de t a i l e d task analysis was begun. "This means,imneeffeet, that the

programmer s t a r t s with the f i n a l c r i t e r i o n task and works backwards,

32

step by step, to the p o i n t where he/she has s p e c i f i e d i n a systematic

f a s h i o n e v e r y t h i n g that the student must go through i n order to reach

the t e r m i n a l behaviour (11:54).

The task a n a l y s i s i n v o l v e s i d e n t i f y i n g the s p e c i f i c responses

that c o n s t i t u t e c r i t e r i o n behaviour, the p a r t i c u l a r c u e - s t i m u l i w i t h

which these responses w i l l become a s s o c i a t e d and the sequencing or

o r g a n i z a t i o n a l requirement of the task to be learned (42:85). The

s p e c i f i c a t i o n of t h i s sequence of d i s c r i m i n a t i v e s t i m u l i and c l a s s of

responses i s most e f f i c i e n t l y accomplished using a f l o w c h a r t . "The

flow c h a r t s p e c i f i e s a l l the d i s c r i m i n a t i v e s t i m u l i to which the l e a r n e r

w i l l have to attend and a l l the responses that he w i l l have to make.

I t serves as a l o g i c a l device f o r f o r c i n g the programmer to t h i n k about

l e a r n i n g from the p o i n t of view of the l e a r n e r and as a source of

prompts f o r frame w r i t i n g " (11:54).

The t h i r d step i n program c o n s t r u c t i o n was to arrange the

subject matter i n t o a l o g i c a l order which i s conducive to r a p i d l e a r n -

i n g and good r e t e n t i o n . The flow chart provided the b a s i s f o r d e c i d i n g

upon the optimum l e a r n i n g sequences and appropriate teaching s t r a t e g i e s .

The arrangement of the subject matter i n t o a l o g i c a l order can be f u r t h e r analysed i n t o three d i s t i n c t steps. F i r s t a h i e r a r c h y of the

m a t e r i a l s must be e s t a b l i s h e d so that the l e a r n e r w i l l f i r s t master

elementary s k i l l s which'he w i l l l a t e r use to develop more complex ones.

Second the h i e r a r c h y of m a t e r i a l s must be composed of steps s m a l l

enough to be taken r e a d i l y by the l e a r n e r without being so s m a l l as

to impede l e a r n i n g . F i n a l l y the program must provide f o r s u f f i c i e n t

33

c o n d i t i o n i n g at each step i n order to be sure that each step w i l l be

adequately learned (7:550).

Upon completion of the task analyses and the sequencing of the

subject matter, the s p e c i f i c frames were w r i t t e n . The concept of a

technology of education introduces the i d e a of the frame as a b e h a v i o u r a l

u n i t , r ather than a simple u n i t of p r e s e n t a t i o n . As a consequence the

frame has a very d e f i n i t e f u n c t i o n to perform and i n c a r r y i n g out t h i s

f u n c t i o n i t must perf o r c e f u l f i l a number of b e h a v i o u r a l requirements.

"Thus, each frame must i n v o l v e a predetermined and p r e d i c t e d change

i n the behaviour of the student working through the sequence and the

change must a l s o be r e l e v a n t and meaningful to the s u c c e s s f u l meeting

of the o b j e c t i v e s " (20:103). In the branching model a frame contains

a concept of i n f o r m a t i o n and a m u l t i p l e choice question to t e s t the

student's understanding of the concept j u s t presented. From the l i s t

of p o s s i b l e answers provided, the student s e l e c t s h i s choice and f o l l o w s

the r o u t i n g d i r e c t i o n s beside t h i s answer. A wrong answer frame e x p l a i n s

the probable cause of e r r o r and gives a f r e s h e x p l a n a t i o n of the con-

cept. The r i g h t answer frame i n c l u d e s a short r e c a p i t u a t i o n covering

the reasons or thought processes by which the r i g h t answer should have

been obtained and then a new concept i s presented and understanding

t e s t e d .

V a l i d a t i o n of the Booklet

"At any stage of i t s development a program represents the

programmer's assumptions as to what m a t e r i a l the student must have i n

34

order to meet the t e r m i n a l c r i t e r i o n behaviours. As the program i s

developed i t i s teste d to see i f these assumptions are v a l i d " (2:244).

The e x t e r n a l c h a r a c t e r i s t i c s of the program were t e s t e d i n two ways,

through s u b j e c t i v e evaluations of subje c t matter s p e c i a l i s t s and a

programming expert and through a v a l i d a t i o n process. This i s an

e s s e n t i a l c h a r a c t e r i s t i c of programmed m a t e r i a l s ; they must be e m p i r i -

c a l l y t e s t e d through a v a l i d a t i o n process, and modified on the b a s i s of

the l e a r n e r ' s responses. The purpose of the process i s twofold; f i r s t l y

i t p o i nts out weaknesses i n the programmandnprpv-idesedaliatonowhirGhcto^

base r e v i s i o n s and improvements, and secondly, i t determines the e f f e c t i v e -

ness of the pro grammas aa teaehihgninstcumen'tnfofothehstuderitstfofowhom -i t was designed to i n s t r u c t .

The v a l i d a t i o n process i s of concern to a l l programmers, but,

un f o r t u n a t e l y , the l i t e r a t u r e on the t o p i c i s o f t e n ambiguous and

va r y i n g p o i n t s of view are f r e q u e n t l y found. Markle e x p l a i n s the con-

f u s i o n by s t a t i n g , "There are no f i r m r u l e s . Each programmer i s on

h i s own." (20:140).

The-testing procedure used f o r the booklet Methods of Adul t

Education f o l l o w s the plan l a i d out by Brethower and i s represented

.schematically i n Figure I I I .

Developmental Test

The purpose of the developmental t e s t i s to check how f a r the

f i r s t v e r s i o n of the programmp.roducedei'slsinifacfectsuitablelfo fothose

f o r whom i t i s w r i t t e n . I t i n v o l v e s g i v i n g the program to a sampling

35

major minor No Problems

I n i t i a l I n d i v i d u a l -> R e v i s i o n > Group P r i n t

Write Tryout Tryout_

Problems

Developmental

Test

F i e l d

Test

Figure I I I . The V a l i d a t i o n Process

of the students, one at a time, i n order to discover any se r i o u s defects

i n the programmetot.revealawhibhclf rameseneede>evisiehsnandndotdetermine"

whether more or l e s s frames are r e q u i r e d to make the programme an

e f f e c t i v e teaching instrument.

The developmental t e s t group was comprised of twelve students

e n r o l l e d i n Education 412 during the 1974 summer s e s s i o n at The U n i v e r s i t y

of B r i t i s h Columbia.aniTheesitudehitstwereasedectedr-randomlyafromuaent

totall'Situdent enrollmenttbd: 2'2utakingTithescou-rses WrTbeyswe

3 6

the programmer and the student discussed i n d e t a i l problem areas i n

the b o o k l e t , student e r r o r s and t h e i r probable cause, and the student's

o p i n i o n of the programme.ThfeeeproductstwereranaiLysedefromoeaehcstuderit,

the programmer's notes on the students' feedback, the v a l i d a t i o n progress

Chartg, and the pre and post c r i t e r i o n t e s t scores. The booklet was

adjusted many times during the developmental t e s t i n g procedure between

the t r y - o u t s w i t h the i n d i v i d u a l students, based on the a n a l y s i s and

i n t e r p r e t a t i o n of the data. I n t h i s manner m a t e r i a l s which were

ambiguous were c l a r i f i e d , b e t t e r examples given and steps that were too

la r g e were broken down i n t o s e v e r a l smaller steps. A very marked taper-

i n g o f f i n the discovery of new weaknesses or problem areas i n the

program was noted as the developmental t e s t i n g proceeded.

F i e l d Tests

Fo l l o w i n g the development t e s t procedure, and r e v i s i o n s , the

program was f i e l d t e s t e d . The purpose of the f i e l d t e s t i s to assess

whether the program s a t i s f a c t o r i l y achieves i t s s t a t e d o b j e c t i v e s when

i t i s used w i t h those f o r whom i t was w r i t t e n under c o n d i t i o n s i n which

i t i s l i k e l y to be used i n p r a c t i c e . The f i r s t f i e l d t e s t group was

comprised of ten students e n r o l l e d i n Education 412 during the 1974

summer s e s s i o n and were s e l e c t e d randomly from the t o t a l student popu-

l a t i o n of 22 t a k i n g the course, the other students having been random-

ly'rassigriedefiopthetdeveiLopmental testin'gggroup . ThT-he tsit'ud'etfts'ewere t y p i c a l of the tar g e t p o p u l a t i o n of students f o r whom the program was

w r i t t e n , were at a p o i n t i n t h e i r course work where the subject would

37

normally be taught, had the d e s i r e d pre-entry requirements, and d i d

not know the m a t e r i a l being taught. A l l students completed a pre-

t e s t and were then given the program to work through p r i v a t e l y . The

main measures c o l l e c t e d from each student were the pre and post

c r i t e r i o n t e s t scores, and the students' o p i n i o n of the program.

The data a n a l y s i s from the f i r s t f i e l d t e s t i n d i c a t e d the need

f o r f u r t h e r major r e v i s i o n s i f the program was to meet the standards

of an e f f e c t i v e teaching instrument. An a r t i s t was h i r e d to provide

i l l u s t r a t i o n s of the major concepts presented i n the b o o k l e t , frame

sequencing was reordered and problem frames were r e v i s e d . A t a b l e

of s p e c i f i c a t i o n s revealed weaknesses i n the c r i t e r i o n t e s t and a new

format was constructed.(Appendix E).

In March, 1975 the program underwent a second f i e l d t e s t . The

subjects were f o r t y a d u l t education students e n r o l l e d i n two courses

of Education 412 o f f e r e d d u r ing the w i n t e r s e s s i o n at The U n i v e r s i t y

of B r i t i s h Columbia. The experimental design used i n the f i e l d t e s t

i s shown' Magi-gu^eilVt.lly(S.eeFals!o-eAp.pTendix B, pp. 154-55).

Group I (n=20) Group I I (n=20)

Test time I c r i t e r i o n t e s t c r i t e r i o n t e s t

Test time I I (one week l a t e r ) c r i t e r i o n t e s t program &

38

The f o r t y students were t y p i c a l of the ta r g e t p o p u l a t i o n of

students f o r whom the program was developed, were at a p o i n t i n t h e i r

course work where the subject would normally be taught, had the designated

pre-entry requirements and d i d not know the m a t e r i a l being taught. The

program was presented as r e q u i r e d course reading and worked through by

the students at home. Both groups provided two c r i t e r i o n t e s t scores,

obtained a week apart. In a d d i t i o n , Group I I recorded t h e i r progress

through the booklet by no t i n g t h e i r response to each item. Their

completion time f o r the program was als o obtained.

The data a n a l y s i s from the second f i e l d t e s t i n d i c a t e d that the

program met the c r i t e r i a of an e f f e c t i v e teaching instrument. A v a l i d a -

t i o n statement, (Appendix B), was prepared to accompany the booklet.

" I n g e n e r a l , a v a l i d a t i o n statement traces the h i s t o r y of a program

through i t s research, development, and f i e l d t e s t s t a t e s , and defines

i t s e d u c a t i o n a l i n t e n t , i n d i c a t i n g w i t h what groups and under what

co n d i t i o n s the program has been shown to work" (34:210).

S p e c i f i c a l l y the V a l i d a t i o n Statement contains the f o l l o w i n g

i n f o r m a t i o n : (1) the b e h a v i o u r a l o b j e c t i v e s , (2) a d e s c r i p t i o n of the

tar g e t p o p u l a t i o n f o r whom the program i s intended, (3) a d e s c r i p t i o n

of the t e s t p o p u l a t i o n s , (4) r e s u l t s of the f i e l d t e s t s to i l l u s t r a t e

the program's performance as a teaching instrument, (5) author q u a l i f i -

c a t i o n s , a l l consultants l i s t e d , (6) average student completion time and

(7) suggestions f o r program use.

"KAbasiic premise which u n d e r l i e s the v a l i d a t i o n statement i s that

the p o t e n t i a l c o n t r i b u t i o n of any i n s t r u c t i o n a l procedure can be best

39

r e a l i z e d i f those who may use i t have adequate i n f o r m a t i o n concerning

the outcomes which i t s use may be counted on to produce" (27:2).

Data A n a l y s i s

The instruments used f o r data c o l l e c t i o n were the c r i t e r i o n

referenced pre and post t e s t s (Appendix E) and the v a l i d a t i o n progress

chart (Appendix C). The i n t e r n a l consistency of the c r i t e r i o n t e s t

was assessed using the p o i n t b i - s e r l a l c o r r e l a t i o n to o b t a i n an i n t e r -

item a n a l y s i s r e l i a b i l i t y and a c o r r e l a t i o n between each item score

and the t o t a l t e s t score. The Kuder-Richardson Formula 20 was a p p l i e d

to determine the o v e r a l l r e l i a b i l i t y score of the t e s t on the b a s i s

of i n t e r n a l consistency. The data was coded onto mark sense cards

and the m u l t i p l e choice s c o r i n g program ( U n i v e r s i t y of B r i t i s h Columbia

MC Score, U n i v e r s i t y of B r i t i s h Columbia Computing Centre, March, 1975)

was used f o r the a n a l y s i s . Two a d m i n i s t r a t i o n s of the c r i t e r i o n t e s t ,

u sing the t e s t - r e t e s t over time procedure, was used to measure f o r

t o t a l t e s t r e l i a b i l i t y . An item d i f f i c u l t y index, and a complete

response p a t t e r n t o the m u l t i p l e choice questions i n P a r t B of the

c r i t e r i o n t e s t was obtained using the SSLE LRNZ Lertap program,

U n i v e r s i t y of B r i t i s h Columbia computing centre.

The scores on the pre and post t e s t s were examined f o r evidence

of a gain i n achievement. This was assessed f o l l o w i n g the developmental

t e s t and f i r s t f i e l d t e s t by r e l a t i n g the t e s t scores obtained a f t e r

the student completed the program to the l e a r n e r ' s i n i t i a l l e v e l of

knowledge as determined by pre t e s t . The measures employed were

40

the t - t e s t and the gain score index (both crude gain and r a t i o of

crude gain to t o t a l p o s s i b l e gain were c a l c u l a t e d ) .

The a n a l y s i s of variance was a p p l i e d to the t e s t scores of the

c o n t r o l and experimental group f o r the second f i e l d t e s t to determine

the s i g n i f i c a n c e of the e f f e c t of the program. This data was key-

punched onto computer cards and the A n a l y s i s of Variance and Co-

variance program was used f o r the a n a l y s i s (author, Malcolm G r e i g , U.B.C.

Anovarrprogramiji,UUB3CC. comp.utinggcentreiy,October?, 19.74'')). THeeDuncans

M u l t i p l e Range t e s t was computed to determine which of the c e l l s of

the experimental design was r e s p o n s i b l e f o r the vari a n c e found to be

s i g n i f i c a n t by the a n a l y s i s of vari a n c e .

A matrice of e r r o r s , based on student responses on the post

c r i t e r i o n t e s t was drawn up f o r the developmental t e s t group and the

f i e l d t e s t groups to p o i n t out areas of weakness i n the program and i n

the t e s t . I n a d d i t i o n , the 90/90 standard, that i s , 90% of the student's

scores reached 90% or more on the post t e s t , was a p p l i e d as evidence of

the teaching e f f e c t i v e n e s s of the program (11, 20).

The v a l i d a t i o n progress charts from the developmental t e s t group

and experimental group i n the second f i e l d t e s t were examined to deter-

mine student progress through the program, and the c o l l e c t i v e student

e r r o r r a t e was computed. A matrice of e r r o r s f o r students and frames

was drawn up to r e v e a l areas i n the program needing r e v i s i o n . The

average student completion time f o r the booklet was determined using

a mean score of t o t a l times a v a i l a b l e .

Chapter IV

ANALYSIS OF THE DATA

The C r i t e r i o n Test

The main instrument used f o r data c o l l e c t i o n was a c r i t e r i o n

referenced pre and post t e s t constructed to measure the achievement of

the s p e c i f i c o b j e c t i v e s of the booklet.

The c r i t e r i o n t e s t used during the developmental t e s t and f i r s t

f i e l d t e s t c o n s i s t e d of ten short answer questions (Appendix D). A l l

students completed the c r i t e r i o n t e s t before beginning study of the

boo k l e t , and again when f i n i s h e d . No estimate of r e l i a b i l i t y was made

of the c r i t e r i o n t e s t .

Data c o l l e c t e d through completion of the c r i t e r i o n t e s t revealed

r e v i s i o n s were necessary. A t a b l e of s p e c i f i c a t i o n s was devised based

on the content of the booklet and i n d i c a t e d a need to i n c l u d e more items

i n the t e s t to be more r e p r e s e n t a t i v e of the t o t a l i n f o r m a t i o n taught.

I t was decided the f i r s t three questions would serve a more u s e f u l pur-

pose as a s e l f - c h e c k t e s t by the students at a r e l e v a n t p o i n t i n the

program and that the diagrams i n question 7 were i n a p p r o p r i a t e to the

nature of the l e a r n i n g content.

The second c r i t e r i o n t e s t (Appendix E) c o n s i s t e d of s i x short

answer questions and twenty-five m u l t i p l e choice questions. The s i x

short answer questions i n Part A are objectively,eeas.ilyaaridrEeli.ably

41

42

scored. I n c o r r e c t responses and blanks were given no score. A score

of one was given f o r each c o r r e c t response except f o r items one and s i x .

The t o t a l p o s s i b l e score f o r item one was four points,aand f o r item s i x

e i g h t p o i n t s (two p o i n t s f o r each s e c t i o n ) . The twenty-five m u l t i p l e

choice items contained a minimum of three and a maximum of four a l t e r n a -

t i v e s . The t o t a l p o s s i b l e score was f i f t y , c o n s i s t i n g of P a r t A-25,and

Par t B-25.

The instrument appeared to have content v a l i d i t y when the items

were judged f o r t h e i r representativeness and relevance i n measuring the

s t a t e d o b j e c t i v e s . This was assessed by the p r o f e s s i o n a l judgment of

two subject matter experts and based on the devised t a b l e of s p e c i f i c a -

t i o n s . ThewconstructnvaOlidiliynof ether instrument nwashnotoexamMedvai >.<

e x t e n s i v e l y , however, c o n f i r m a t i o n of the teaching e f f e c t i v e n e s s 6*f the

program through the experimental design would tend to i n d i c a t e that the

instrument possessed some degree of construct v a l i d i t y .

The r e l i a b i l i t y of the t o t a l t e s t was assessed using t h e . t - t e s t

and a summary of r e s u l t s i s shown i n Table 3.

Table 3

Suinma.ryyoifResuiisso.fft?*Testf'forCeriterd!onTfEest

Time 1 -x Tri.uie 2, Tr Time 2 x d f E-value P

13. 7 15.45 19 34 .262

The 'E-'value was s i g n i f Icantc.totthe ir.'OS l e v e l e o f l s i g n i f i c a n c e . The t e s t scores f o r the second t e s t time were s l i g h t l y higher which i s a

normal r e g r e s s i o n toward the mean and the e f f e c t s of having taken the

43

same t e s t twice. However, the r e g r e s s i o n was not s t a t i s t i c a l l y s i g n i -

f i c a n t . The o v e r a l l r e l i a b i l i t y o f the t o t a l t e s t was, t h e r e f o r e ,

considered s a t i s f a c t o r y . The Kuder Richardson Formula 20 was a p p l i e d

to estimate the r e l i a b i l i t y on the b a s i s of i n t e r n a l consistency. The

r e s u l t s are shown i n Table 4.

Table 4 Summary of Results using the Kuder Richardson Formula 20

Standard Standard Number of items Mean E r r o r D e v i a t i o n R e l i a b i l i t y

43 12.825 .677 4.284 60

Thus the c r i t e r i o n t e s t was accepted as a r e l i a b l e instrument

f o r the purposes of t h i s study.

An item a n a l y s i s was shown,, .by applying the p o i n t b i - s e r i a l

c o r r e l a t i o n t o the scores of the two groups obtained at t e s t time I .

This c o r r e l a t i o n i s used when one dichotomous v a r i a b l e i s c o r r e l a t e d

w i t h the continuous v a r i a b l e and i s a shorteruversronoofofhehprpduct-st

moment c o r r e l a t i o n formula. The r e s u l t s showed t h a t , although the t o t a l

t e s t was r e l i a b l e , m u l t i p l e choice items 4, 9, 12, 15, 22, and 23 f a i l e d to meet an acceptable l e v e l of r e l i a b i l i t y . Therefore, i t i s recommended

that f o r f u t u r e use of the program these items be reexamined and deleted

or r e v i s e d .

An item d i f f i c u l t y index, w