SOLO, RASCH, QUEST and Curriculum Evaluation S. Alagumalai Flinders University of South Australia The Structure of the Observed Learning Outcome (SOLO) taxonomy describes the growth in performance in many learning tasks, beginning from task engagement to expertise level. On the other hand, the Rasch analysis could provide greater information available from several questions per level to achieve a more complete understanding of a learner's progress in a particular topic or subject. This paper attempts to review curriculum evaluation and modification from the perspective of the SOLO taxonomy and the Rasch model, with the help of the QUEST software. It is believed that the SOLO taxonomy, coupled with the Rasch model and QUEST software, would enable teachers to specify levels of competence in particular sections in a topic that is generalisable across students, topics and levels, which makes possible meaningful curriculum evaluation and modification. Introduction There is a large amount of research evidence to indicate that most students complete higher education with little surface declarative knowledge of their disciplines and that they do not learn to think like experts in their areas of study (Ramsden, 1988). In line with this, "a fundamental reason why students fail to learn and develop intellectually is that the questions or problems directed at them are of an inappropriate level or, conversely, that students lack the appropriate conceptual level, strategy or skill necessary to cope with the problems and questions under investigation" (Courtney, 1986). This applies likewise to the specific instructional objectives (SIO) available in a curriculum. Thus, apart from simply assigning 'pass' or 'fail' grades to students in a test or an examination, which could be linked explicitly to an SIO (Oliver, 1977: 102), it is important to evaluate the quality of learning and thus the teaching itself. In a central curriculum environment, teaching is heavily dependent on the curriculum content and its assessment (Courtney, 1986). The curriculum interacts with teachers and students in complex and important ways. The curriculum is visualised as an "operational plan that includes the substantive content, the expected actions and behaviours of teachers, the expected actions and behaviours of the students, and the technology (explicit pedagogic strategies) for conveying subject matter and structuring teacher and student activities" Murnane & Raizen, 1988: 120). Underlying the dynamic evolution of any curricula is the assessment of the curriculum itself (Denis et al., 1976: 50-56; McCormick, & James, 1988: 2-3). Though
Transcript
SOLO, RASCH, QUEST and Curriculum Evaluation
S. AlagumalaiFlinders University of South Australia
The Structure of the Observed Learning Outcome (SOLO) taxonomy describes the growth in performance in many learning tasks, beginning from task engagement to expertise level. On the other hand, the Rasch analysis could provide greater information available from several questions per level to achieve a more complete understanding of a learner's progress in a particular topic or subject. This paper attempts to review curriculum evaluation and modification from the perspective of the SOLO taxonomy and the Rasch model, with the help of the QUEST software. It is believed that the SOLO taxonomy, coupled with the Rasch model and QUEST software, would enable teachers to specify levels of competence in particular sections in a topic that is generalisable across students, topics and levels, which makes possible meaningful curriculum evaluation and modification.
Introduction
There is a large amount of research evidence to indicate that most students complete higher education with little surface declarative knowledge of their disciplines and that they do not learn to think like experts in their areas of study (Ramsden, 1988). In line with this, "a fundamental reason why students fail to learn and develop intellectually is that the questions or problems directed at them are of an inappropriate level or, conversely, that students lack the appropriate conceptual level, strategy or skill necessary to cope with the problems and questions under investigation" (Courtney, 1986). This applies likewise to the specific instructional objectives (SIO) available in a curriculum. Thus, apart from simply assigning 'pass' or 'fail' grades to students in a test or an examination, which could be linked explicitly to an SIO (Oliver, 1977: 102), it is important to evaluate the quality of learning and thus the teaching itself. In a central curriculum environment, teaching is heavily dependent on the curriculum content and its assessment (Courtney, 1986).
The curriculum interacts with teachers and students in complex and important ways. The curriculum is visualised as an "operational plan that includes the substantive content, the expected actions and behaviours of teachers, the expected actions and behaviours of the students, and the technology (explicit pedagogic strategies) for conveying subject matter and structuring teacher and student activities" Murnane & Raizen, 1988: 120). Underlying the dynamic evolution of any curricula is the assessment of the curriculum itself (Denis et al., 1976: 50-56; McCormick, & James, 1988: 2-3). Though
different models do exist on curriculum evaluation, Raizen & Jones (1985) have indicated that there are, "no indicators to evaluate curriculum quality." Furthermore, there has been no work done in measuring the 'curriculum shift' over a period of time with different student samples. All three dimension of curriculum quality, namely the depth of topic treatment, scientific accuracy and pedagogic quality,
according to Murnane & Raizen (1988: 136) present, "difficult problems of measurement as they do not lend themselves to the kind of detailed analysis for assessing breadth of coverage."
This paper attempts to outline the measurement of curriculum quality, with reference to the depth of topic treatment, and also in attempting to illustrate the measurement of 'curriculum shift'. [Wilson (1988) indicates that the, "presentations of learners' progress/cohort's progress on a logit scale is a potent tool for the analysis of educational program, which is yet to be seen."] These quality 'measurement and control' (hereafter termed curriculum evaluation) facilities of the curriculum has been made possible through the SOLO (Structure of the Observed Learning Outcome) taxonomy and Rasch Latent Trait measurement model. The latter's application has been made simple with the availability of the QUEST (Adams & Khoo, 1994) software. Thus, this papers seeks to outline how the SOLO taxonomy and Rasch model via the use of QUEST enables the meaningful quality evaluation of curriculum and thus in facilitating appropriate evaluation of higher order thinking skills within a curriculum. Such a 'synthesis' of a probalistic framework, via the Rasch model, and the SOLO taxonomy is needed to enable the modeling of student's behaviour to better understand the salient qualities of performances (Mislevy, Yamamoto & Anacker (1991).
The SOLO Taxonomy
The study of cognitive development from birth to maturity indicates a qualitative improvement in the structure of cognitive products with age. This progression from the concrete operational stage to the highest abstraction stage has been outlined clearly by the Piagetian theory. However, Biggs and Collis (1982: 31) indicates that there are clear distinctions between development and learning. They argue that there, "is a fundamental mistake to identify learning responses with responses to developmental stage tasks" (Biggs & Collis, 1982: 31). Thus, to evaluate the quality of learning, Biggs & Collis proposed the Structure of the Observed Learning Outcome (SOLO) theory. The SOLO is an acronym for 'Structure of the Observed Learning Outcomes', and is based on the observation that learners display a consistent sequence, or 'learning cycle', in the way they go about learning them (Biggs & Collis, 1982: 21-23). The SOLO describes five levels which include the
levels of prestructural, unistructural, multistructural, relational and extended abstract are isomorphic to the Piagetian stages but logically distinct from it (Biggs & Collis, 1982: 31). The SOLO levels are based on the structural organisation of the knowledge in question, from incompetence to expertise.
The SOLO taxonomy is a means of evaluating learning in a way readily understood by teachers, students and parents, and is consistent with educational theory and practice. Knowledge acquired through learning may be applied in a variety of ways such as answering a question, solving a problem, making a judgement, evaluating an information or explaining what has been learned. Lewis (1994) contend that if students are operating from one view of learning and teachers from another then
it is likely that the resulting learning environments and outcomes will not be satisfactory. Thus, it is imperative to have a match between the latter two, especially when quality is at stake. Thus it is important to note that the structure of the student's response will reflect the quality of that student's learning because the application of knowledge involves using a set of components (facts, concepts, skills or strategies) that may be used independently or integrated with each other. The evaluation technique proposed by Biggs & Collis (1982:17-31) involves the classification of the response in terms of its structure, i.e. with reference to the SOLO. Table 1 summarises the key concepts of the SOLO taxonomy.
Developmental base stage with minimal ageSOLO description1Capacity2Relating operation3Consistency and closure4Response StructureCue Response
Formal Operations(16+ years)Extended AbstractMaximal: cue + relevant data + interrelations + hypothesesDeduction and induction. Can generalise to situations not experiencedInconsistencies resolved. No felt need to give closed decisions - conclusions held open, or qualified to allow logically possible alternatives.(R1, R2, or R3)
Concrete Generalisation(13 - 15 years)RelationalHigh: cue + relevant data + interrelationsInduction. Can generalise with given or experienced context using related aspectsNo inconsistency within the given system, but since closure is unique so inconsistencies may occur when he/she goes outside the system
Middle Concrete(10 - 12 years)MultistructuralMedium: cue + isolated relevant dataCan "generalise" only in terms of a few limited and independent aspectsAlthough has a feeling for consistency can be inconsistent because closes too soon on basis of isolated fixations on data, and so can come
to different conclusions with same data
Early Concrete(7 - 9 years)UnistructuralLow: cue + one relevant datumCan "generalise" only in terms of one aspectNo felt need for consistency, thus closes too quickly: jumps to conclusions on one aspect, and so can be very inconsistent
Pre-operational(4 - 6 years)PrestructuralMinimal: cue and response confusedDenial, tautology, transduction. Bound to specificsNo felt need for consistency. Closes without even seeing the problem.
Kinds of data used: X = irrelevant or inappropriate; ( = related and given in display; ( = related and hypothetical, not given
Table 1. SOLO Stages of Cognitive Development (Biggs & Collis, 1982: 24)
The level of abstraction that a learner uses when handling the elements of a task is called the 'mode' of that learning episode. Change brought about by cognitive development occurs when the highest mode that a learner can use with any task increases with age; change brought about by learning occurs when the student's responses become more complex within the learning cycle of that task. The 'target mode' is that level of abstraction that is appropriate to a lesson, and that at which the teacher can expect the students to engage the task. Table 2 summarises the various modes and levels (Biggs & Collis, 1989: 152).
Mode*Structural Level (SOLO)
Next5.Extended abstract. The learner now generalises the structure to take in new and more abstract features, representing a higher mode of operation.
Target4.Relational. The learner now integrates the parts with each other, so that the whole has a coherent structure and meaning.5.Multistructural. The learner pick up more and more relevant or
correct features, but does not integrate them.6.Unistructural. The learner focuses on the relevant domian, and picks up one aspect to work with.
Previous1.Prestructural. The task is engaged, but the learner is distracted or misled by an irrelevant aspect belonging to a previous stage or mode.
[*Modes are levels of abstraction, progressing from concrete actions to abstract concepts and principles, corresponding in large part to the developmental stages referred to by Piaget (1950)][Piaget (1950): sensori-motor ( ikonic ( concrete-symbolic ( formal-1 ( formal-2]
Table 2. Modes and Levels in SOLO Taxonomy (Biggs & Collis, 1989: 152)
The SOLO taxonomy, when applied appropriately, allows for higher-order responses, and teachers must be 'ready' for divergent answers. The task
at hand is the categorisation of the responses to the appropriate SOLO level. For example, if a student gives a 'higher-order' answer to a simple 'basic' question, different raters/teachers may respond differently. Some may regard the answer as 'irrelevant', other may indicate that there is 'a modal shift' and a few may even indicate that the response did not 'meet the teacher's objectives'.
Hence, "an extended abstract response may either be a bonus or an irrelevance. In closed subjects like science and mathematics, the extended abstract response frequently results in what is referred to as an 'elegant' solution, as opposed to a merely 'correct' one at the relational level. In other subjects, however, an extended abstract response may lead the student into unexpected or divergent directions, the relevance of which some might question. Those direction may, however, foreshadow future potential, and may become part of the target mode in subsequent instruction at a higher level of abstraction" (Biggs & Collis, 1989).
Furthermore, the target modes vary considerably from primary, through secondary, to tertiary levels of education. The unistructural-multistructural-relational learning cycle repeats itself within all modes. By carefully defining the appropriate level of abstraction of the target mode, it becomes possible to specify desirable curriculum objectives and performance outcomes throughout school, according to the desired SOLO level within the appropriate mode. Figure 1 aptly captures this (Biggs & Collis, 1989: 160).
Figure 1. SOLO Stages and Curriculum Goals (Biggs & Collis, 1989: 160)
Courtney (1986) indicates that the principle underlying the SOLO taxonomy should be used for both formative and summative aspects of
evaluation in schools. He indicates that this would facilitate the writing of examines report which can be easily apprehended by teachers, students and parents. Along similar lines, it can be argued that the feedback based on the SOLO taxonmoy would be useful for curriculum planners and policy makers and thus in curriculum and programme evaluation.
The strength of the SOLO taxonomy as a tool for evaluating is based upon the assumption that it is, "structurally organised and discriminates well-learned from poorly-learned material in a way not unlike that in which mature thought is distinguished from immature thoughts" (Biggs & Collis, 1982: xi). They also propose a general model that links a cyclical growth in the complexity of learning to the stages of cognitive development outlined in neo-Piagetian theor and can
be linked to the cyclical curriculum development as postualated by the 'spiral curriculum' model.
However, the SOLO taxonomy departs from the Piagetian model, which focuses upon the developmental stages identified through responses to tasks embodying logical and mathematical concepts. The focus of the SOLO taxonomy however, is on quality of learning which is determined by the response to task material drawn from school subjects and curriculum content. They purport a clear demarcation between learning and development in these models.
The SOLO taxonomy thus, has implications for the classroom teacher and the assessment board. In constructing a/an test/examination, it would be unfair if only particular levels are addressed, i.e. students operating at a particular level are disadvantaged to answer questions at a higher level. To overcome this, open questions could be posed and the responses categorised accordingly (details of this has been reported in Keeves & Alagumalai, 1996).
An alternative practical solution would be to set a series of superitems (Cureton, 1965). Here, superitem refers to a common situation or information stem, followed by four questions, one at each structural level of the SOLO taxonomy. Numerous studies using the superitem concept have been attempted over the past few years (Collis & Romberg, 1981; Courtney, 1986; Wilson, 1985; Wilson, 1989). One significant feature of these superitem tests is that it is capable of showing in an objective way the significant movement up the SOLO levels over the years of schooling. The example below illustrates this (using hypothetical data, adapted from Courtney, (1986)):
Table 3. 'Shift' in SOLO levels with years of schooling
Thus, such superitems based on the SOLO taxonomy would be useful for both formative and summative evaluation and in identifying 'shifts' in learning. Furthermore, it would be of great value for curriculum planners to evaluate the effectiveness and appropriateness of the topics/SIO planned for. In retrospect it enables the measurement of the 'depth' of the curriculum coverage. Hence, educational objectives can be measured against performance and thus facilitate curriculum
evaluation. As indicated by Courtney (1986: 50), such evaluation would enable one to move away from norm-based model of evaluation to a criterion-based one. This is facilitated by the use of Rasch probabilistic model, and will be discussed in the next section.
Biggs & Collis (1989) have indicated that successful implementation of school-based curriculum requires that teachers and curriculum
developers use a common metric to refer to desirable levels of attainment across the curriculum and across year levels, and in terms of which learning outcomes can be criterion-referenced. Thus, the SOLO taxonomy is essentially a language for describing the level and quality of learning both within and across the curriculum.
In all these items (both superitems and others), the level of response depends to a large extent upon the kind of questions posed (Courtney, 1986). The response to a particular question would be indicative of the learner's location on a latent dimension (Wilson, 1989). However, Wilson (1989) cautions that in interpreting these responses, one needs to go beyond the classification of responses into discrete classes and thus allow for a probabilistic interpretation mapping response patterns into levels of the SOLO taxonomy. He further argues that the use of latent trait interpretation does not contradict the cumulative and hierarchial nature of the SOLO taxonomy and discusses inherent advantages elsewhere (Wilson, 1989; 131).
Thus, to derive meaningful interpretation from the responses, there is a need for a, "strong response function that satisfies important theoretical criteria, and which allows for the detection and interpretation of response patterns, both expected and deviations from it" (Wilson, 1989: 131). This leads to the use of the Rasch model and is discussed next.
The Rasch Model and Item-Response Theory
Research in the past decade have increasingly criticised developmental tests that have been constructed based on classical psychometric theory. This has resulted in the identification of inadequacies in existing norm-referenced developmental measures (Rasch, 1960; Andrich, 1988; Archbald & Newmann, 1988: 52-59). Classical psychometric theory, also called Classical Test Theory (CTT), is only one of the three primary measurement models, the other two being Generalisability Theory (GT) and Item Response Theory (IRT). The IRT is based on a scaling model proposed by Rasch (1960, 1966, 1980).
IRT has become a very popular topic for research in the measurement field (Hambleton, 1982). He (1982: 2) indicates that the interest in IRT models stem from two features which are obtained when an item response model fits a test data set:descriptors of test items (item statistics) are not dependent upon the choice of examinees from population of examinees for whom the test items are intended; andthe expected examinee ability scores do not depend upon the particular choice of items from the total pool of test items to which the item
response model has been applied.Thus, the "invariant item and examinee ability parameter, as they are called, are of immense value to the measurement specialists" (Hambleton, 1982: 2). This implies that independent random sampling of items that have been calibrated using the Rasch model will yield equivalent ability estimates for any individual or group from the population. Furthermore, any sample from the specified population will yield equivalent item difficulty estimates. This seems attractive, but it must be admitted that IRT is not free from controversies, but it is adequate at this point of time to serve a variety of uses (Lord, 1980; Hambleton, 1982).
The Rasch model (Rasch, 1960, 1966, 1980) for the analysis of test data provides a way to place learners and questions on a scale with a clear probabilistic interpretation of distance on the scale. In addition to its simplicity, both in theory and in application, it is the only latent trait model which enables sample-free estimation of person as well as item-parameters.
The Rasch model is a mathematical formulation linking the probability of the outcome when a single person attempts a single item to the characteristics of the person and the item. The model assumes that item are dichotomously scored, test is not speeded and the chances for success can be defined as a function of the ratio of a person ability ((() to item difficulty ((i):
C(i = (( / (i [1]
where C(i is the chance for person ( to succeed on item i, (( is the ability of person (, and (i is the difficulty of item i. If the probability of obtaining a correct response is defined as the chance for success divided by one plus this chance, then
P(i = C(i / (1 + C(i ) = ((( / (i ) / (1 + (( / (i )[2]
The equation [2] is an expression for the Rasch probability of a correct response in terms of only two parameters, person ability and item difficulty. To make the model simpler by putting it into an additive form, we define (( as the log ability of person ( and (i as the log difficulty of item i. Thus, the probability of a correct response can be expressed as:
P(i = [e ((( - (i )] / [1 + e ((( - (i )][3]
It is evident from equation [3] that the probability of success is greater than 0.5 when (( > (i , i.e. the person ( has more latent ability than item i requires. Similarly, when the item is too difficult, (( < (i and the probability is less than 0.5. [Please refer to Wright & Stone (1979) for further details about the derivation and interpretation of Equation [3].]
In contrast to other latent trait models, the Rasch model specifies only one item parameter, difficulty. Other models exist (2-parameter logistic, 3-parameter logistic, Rating Scale, Partial Credit, etc.. please see Lord 1980; Hambleton, 1982; Keeves & Alagumalai, 1996), and
these models specify additional item parameters of discrimination and tendency to provoke guessing. The Rasch model essentially sets the guessing parameter to zero and treats the discrimination parameter as it were constant for all items. These are strong assumptions, but have been supported by work done by Wright (1977). Because of these assumptions, "the Rasch model is the only latent trait model for a dichotomous response that is consistent with 'number right' scoring" (Wright, 1977). As some sections of any examination have items scored dichotomously, the Rasch model would be useful in identifying the criterion of any latent measure and thus aids in quality measurement. Wright (1977) further adds that it is the only method for both obtaining estaimates of item parameters free of the ability distribution of the person sample and estimate of person parameters free of the difficulty distribution of the item sample. Along similar lines of argument, Hambleton et al., (1978) have suggested that the Rasch model may hold promise for examining test bias and individual item bias. These applications of the Rasch model, allows for evaluating the curriculum, via the SIOs linked to specific test items, and thus measure both the quality of the SIOs in question and also the learning 'shifts; over period of time and with sample.
The logit, a unit used in the Rasch model to describe item difficulty and person ability, is mathematically and computationally convenient (Choppin, 1983). The Rasch model compares ability and difficulty scores on a natural log scale (with reference to equation [3]) on which item difficulty and person ability may attain any value whatsoever. A logarithmic scale is used because it enables the interaction between item difficulty and person ability to be described in an additive way. In reference to this logit unit, Choppin (1983) indicates that a change in achievement of one logit represents a considerable amount of learning. In reality, in a given subject area, an average child's achievement would rise by rather less than 'half a logit' in a 'typical school year' (Choppin, 1983). This is an important indicator to measuring both quality and 'shifts' in learning within a given curriculum, especially when a central curriculum is in place. Apart from measuring curricula quality, the Rasch model has been traditionally used in item banking, test analysis, adaptive testing and subject assessment. As test analysis and subject assessment is pertinent to the discussion of curriculum evaluation in this paper, they will be examined in detail next.
Test Analysis. The Rasch analysis provides item statistics that are useful in evaluating the appropriateness of developmental assessment
measures and the items which compromise such measure. The following information can be obtained through the Rasch analysis (Andrich, 1988; Snyder & Sheehan, 1992):the degree to which item-response data on a specific test are unidimensional for a given population;the congruence between traditional item statistics and the Rasch calibrated equivalent (item difficulty statistics);variance/invariance of Rasch item parameters across subgroups of a population;the ability of a test to discriminate between distinct ability groups and categories within a population;the degree to which the items in a test represent distinct difficulty
levels or the SOLO levels;the reliability and error associated with each ability score and each item difficulty value;the degree of difficulty of the test for a subgroup within a population or to the population as a whole;the degree to which the item response patterns (through the use of ICCs) for a subgroup exhibit more variability than expected; the identification of items that are excluded from Rasch calibration for a particular population of respondents due to facility, difficulty or discrepancy of response patterns from the expected model;the identification of item bias; andthe relative distance between of 'anchored tests'.
Generally, the Rasch model enables the identification of topic 'clusters' within a subject area and also in identifying 'easy' and 'difficult' sections within a topic of interest. The advantage of using the Rasch model is, as mentioned earlier, is that the items or associated SIOs in question is subject to scrutiny devoid of the sample who attempted it. The total number of respondents could also be used as a gauge of how the trend/shift is over the years, especially when selected items are anchored across years and also across levels.Subject Assessment. The Rasch analysis yields scores for item difficulty and person ability that are on the same natural log scale. This is useful in generating and applying curriculum-referenced tests and hence in evaluating the curricula-area itself. Anchor items could be used to generate a score for a student that would indicate his/her ability level along a curricula continuum and thus gauge the student's success on items and eventually the SIO the student did not encounter. Comparable work has been done by Keeves & Kotte (1995) and lends support for such a methodology to be used for curriculum evaluation. The information obtained would be useful for:providing appropriate 'target' curricula-area for subgroups to facilitate optimal learning and thus functioning;facilitate level-appropriate formative assessments setting, aimed at the subgroups ability levels;allow time-specific and ability-specific intervention programs to be
placed; allow introduction and monitoring of higher-order thinking skills with a curricula or cross-curricula areas; andin providing the society a quality 'evolving' curriculum.
Snyder & Sheenan (1992) indicate that the modifications in the curriculum could be considered if the item difficulty order varied substantially from the anticipated sequence. Hence, if the revised sequence can be accounted for within a curricula model, modifications may be warranted.
To facilitate all of the above analysis, QUEST (Adams & Khoo, 1994), a computer program has been developed. To illustrate the use of the Rasch model to identify the various SOLO levels, details of the Mathematics instrument used for the International Association for the Evaluation of Educational Achievement (IEA) will be used.
The seven items selected (Table 4) were part of the Mathematics Performance Scale and used in the research study undertaken in 1964 in
12 countries by the IEA (Husen, 1967; Keeves, 1968; Keeves & Bourke, 1976). Australia tested students in 1964 at the lower secondary school stage (Population 1) and the upper secondary school stage mathematics student (Population 3A) (Keeves, 1968; Keeves & Bourke, 1976).Item scored using the Partial Credit Model (Masters, G.N., in Keeves, p. 4302 - 4307) for the following items used in the Australian Mathematics (AUSMATHS - 1964) study:Item 15Item 17Item 32Item 41Item 63Item 64Item 70
SOLO Levels and the assigned Partial Credits:Extended Abstract3 stepsRelational2 stepsMultistructural1 + 1 stepsUnistructural1 stepPrestructural0 step
Table 4. SOLO levels and score assignments for alternativesTo Keeves & Kotte (1995), it is generally possible, "from the alternative responses chosen by students to multiple choice to identify the level of performance in which they are operating in order to make a response to an item." Thus, the alternatives in a multiple choice question can be constructed with the SOLO taxonomy accounting for different levels of operation. Table 4 aptly captures the possible order in constructing alternatives for multiple-choice questions and assigning partial scores based on the SOLO levels. It must be noted that the response to these multiple-choice questions with the framework of the SOLO taxonomy, entails a stochastic rather than deterministic
processes. Consequently, the responses of students to these items involve probabilities with a certain degree of randomness (Keeves & Kotte, 1995), which thus allows for the use of the Rasch model. The QUEST output for this is given below:
LogitsNumber of students (X = 27)Item/ResponseSOLO Levels
The above QUEST output gives valuable information and could be used for understanding the curricula objectives via the test items in detail. The first column indicates the logit scale/metric. The second columns indicates the distribution of the students in the test and positions them by their ability. Corresponding outputs of the QUEST program also gives the mean of the cohort together with the other necessary test statistics (see Adams & Khoo, 1994). The third column indicates the position of the various items, with regards to their alternatives based on the SOLO levels. Work by Keeves & Kotte (1995) indicate that the alternatives which require a relatively higher 'operation' are grouped. Hence, column four captures the approximate SOLO levels for the group of 'related' items in column three.
This, side by side matching of the QUEST output for items with the
respective alternatives against the 'approximate' SOLO levels enables the test constructors to counter-check their assignment of levels (Table 4) and also to verify the distribution statistics of the items in the multiple-choice tests. Items that don't fit the Rasch model can be re-examined for 'flaws'. Furthermore, such a procedure would enable the identification of bias item(s) within selected groups. Thus, such a rigorous method of item analysis could provide a definite base for curriculum evaluation. The next section attempts to bring together all the above methods into curriculum evaluation so as to improve the quality of education and to facilitate objective reporting.
Curriculum Evaluation
Curriculum development is part and parcel of coping with the needs and changes in society. Shane & Tabler (1983: 34-35) outline alternative approaches to curriculum planning and development. They are:The regressive option - return to some of the values and practices that have been discarded.The conservative option - leave things as they are.The liberal option - adopt changes that are mandated by a changing
society.The experimental option - create new educational designs.The regenerative option - adopt new approaches to learning experiences.The eclectic option - any one or any combination of the above might be the best option.
They further chart out taxonomically the curriculum options available to curriculum specialists and policy makers. Figure 2 seeks to extend thinking about curriculum development, change and evaluation and to enable thinking in terms of better learning for living (Shane & Tabler, 1983: 36).
Hence, the very act of deciding on the details of a curriculum requires one to consider the options and evaluate the utility of the alternatives. Evaluation broadly defined is, "the collection and use of information to make decisions about an educational program" (Cronbach, 1963: 672, quoted in McCormick & James, 1983: 172). Oliver (1977: 91) however, cautions us that people working on curriculum evaluation and improvement must, from the start have a concept of the values and directions as to the goals they hope to achieve. They then must carefully formulate and clearly state their objectives (Oliver, 1977: 91).
Figure 2. The Taxonomy of curriculum options (Shane & Tabler, 1983: 36)
An essential part of curriculum development is curriculum evaluation (McCormick & James, 1983; Eisner, 1995: 150) and this lays the basis for the evaluation of quality in education. Curriculum evaluation might provide a mode of accountability, improvement of student learning,
preparation of students for societal needs, enhance quality of education, promote professional development and also facilitates the dynamic evolution of the curriculum in place through the review process. Thus, curriculum evaluation is the process of delineating, obtaining and providing information useful for making decisions and judgements about curricula (Davis, 1981: 49, quoted in McCormick & James, 1983: 172).
Associated with this evaluation process is quality. Wicks (1987: 43) argues that, "quality is an attribute and is measured by outcomes." Though some will argue that measures of quality are more impressionistic rather than specific, Wicks (1987: 43) illustrates that 'proportion and quantity' are indicators of quality.
Any form of decision making requires judgement, and judgement requires
evaluation of choices, options and alternatives (Eisner, 1994: 150). Quality and judgements of quality are central to any evaluation study (Kemmis & Stake, 1988: 7). Wicks (1987: 44) further argues that measures of quality that focus on student performance are diagnostic. These measures can then be used to maintain good curricula/program and improve those which do not meet expectations. Hence, in attempting to quantify performance and in evaluating the curricula, the quality of education as a whole is reviewed. In wanting to accommodate, and even evaluate higher-order skills, the above procedure of using the SOLO taxonomy with the Rasch model is a possible and practical solution.
The next step after the decision to evaluate, is to decide what aspects of curriculum evaluation are likely to provide the most relevant information for decision-making. James (1982, quoted in McCormick & James, 1983: 175) suggests two areas of interest: those that focus on educational aims and antecedents (prior conditions) and outcomes, and those that focus primarily on educational process.
Holt (1981, quoted in McCormick & James, 1983: 175) however disinguishes three key areas in curriculum evaluation: outcome evaluation, procedural evaluation and process evaluation. Many more comparable models exists (McCormick & James, 1983: 175-177), but McCormick & James (1983: 177) argue that Robert Stake's (1967) framework for evaluation, "still provides the most comprehensive answer to the question of what to evaluate." They note that there are important distinctions between description and judgement, two separate but complementary activities of evaluation (further details can be obtained from Kemmis & Stake, 1988: 5-32).
Though Stake (1967, quoted in McCormick & James, 1983: 179) emphasis the evaluation of the 'processes', he does not reject the evaluation of the 'product' of the curriculum. Along similar lines of argument, Kemp (1985: 227) indicates that summative evaluation allows one to reach unbiased and objective answers and then to decide whether a curricula/program is achieving the goals it was originally intended to reach. Oliver (1977: 91) further indicates that any evaluation process has the following ingredients:
It can thus be argued that the Instructional Objectives could be directly evaluated through the 'products' of summative assessments. This is in line with Oliver's (1977: 92) contention that, "once the objectives have been identified, questions can be raised as to what learning activities can be set up to carry them out. Then there must be a process of evaluation to determine whether or not goals have been realised. Both in thinking and in practice, one moves carefully from
one stage to the other."
Oliver (1977: 101-102) states that the objectives will provide a basis for the selection of the learning activities. He further reiterates that objectives are tied to evaluation; evaluation requires a careful formulation of objectives to serve as the 'benchmark' against which to judge the adequacy and appropriateness of materials, methods and activities. Thus, the evaluation of objectives themselves lend credibility to subject-specific curriculum evaluation. Objective (including SIOs) when properly conceived, skilfully presented and objectively evaluated, serve as guides, as directions, as incentives and as criterion for quality control. For these reasons, currriculum developmental theory includes a study of objectives as a major step.
Thus, it can be argued that the evaluation of the 'product' of any curriculum, with respect to formative and summative assessment, through the use of the SOLO taxonomy and Rasch model, could thus be an objective 'first-step' in curriculum evaluation. This is in line with the argument that, "one cannot judge the relative worth of any activity unless he/she judges it against some other criteria (Oliver, 1977: 92). The Rasch model makes it possible through its flexibility in supporting both norm-referenced and criterion-referenced measures. It must be admitted that the evaluation method outlined here is only, but part of a more holistic evaluation of the curriculum.It must also be mentioned that the evaluation process, like any other research methodologies, has to have both validity and reliability so as to allow rationale and consistent decision-making (McCormick & James, 1983: 180-191). These criteria can be ascertained through the use of the Rasch model and thus the first step in curriculum evaluation is an objective and unbiased process in the lead to quality control and improvement.
It is often assumed by persons in education and training that the merits of a curricula or program are obvious to other persons in the institution, organisation or society. Unfortunately, rarely is this conclusion true (Kemp, 1985: 227). Hence, any form of evaluation is a worthless exercise if it is not reported to the significant others. In curriculum evaluation, the nature of the reporting process is important in determining whether the evaluation is primarily for accountability or improving education (McCormick & James, 1983: 321). They further indicate that, "a key to the form of any report is the concept of audience. This should take into account of the division into insiders and outsiders (to the school, including the teachers, curriculum specialists/planner and policy makers), but also consider the audience group and its accountability relationship to the teachers in the school. The parents are the most obvious group to receive a report; not only is this a matter of their rights (contractual accountability) but
teachers have a sense of moral accountability" (p. 321).
With this as a backdrop, the example below would serve to illustrate the 'product' evaluation of a specific curriculum area (Physics), using the SOLO and the Rasch model as discussed in the preceding sections.
Syllabus:GCE Ordinary Level (Physics 5054)Sample Topic:2.2Thermal Properties(c)Heat CapacitySample SIOs:- relate a rise in temperature of a body to an increase in internal energy (random thermal energy)- show understanding of the terms heat capacity and specific heat capacity- describe experiments to measure the specific heat capacity of solids and liquids and make the necessary calculations
1990If the heat energy is removed from an object, its temperature will normallyA.fallD.riseB.fall then riseE.rise than fallC.stay the same
1991The specific heat capacity of copper is 400 J/(kgK). A mass of copper is heated for 40s by a heater which produces 100 J/s. What is the rise in temperature?A.5 KD.50 KB.10 KE.80 KC.20 K
In drawing up the Table-of-Specifications for the summative assessment, test items can be directly 'linked' to the SIOs concerned. Table 5 is a hypothetical output of QUEST for anchored items over selected years.
Table 5. Hypothetical Summary Output (QUEST)
The above hypothetical output illustrates the evaluation of the overall curriculum over the selected years and monitors the curriculum 'shifts'. There has been relatively more gains at the 'lower' SOLO
levels for a particular item and could reflect that the content of the item(s) concerned could be 'common knowledge'. The breadth and depth of content coverage for that SIO could be expanded. (A decade ago only final year Physics and Electronic Engineering students could 'handle' a
computer - look at the kind of arcade games the kids could handle now!). More general instructional objectives and related objectives at the next level (GCE Advanced Level) could be incorporated to enhance the quality. They could even be a move to introduce 'topical' rather than having to deal with SIOs (Eraut, 1989: 341-352). Furthermore, the percentage distribution/quota of different-level questions (with regards to knowledge, application etc..) in an examination could also be adjusted accordingly so as to elicit higher-order thinking in subsequent cohorts of students. This would mean that by merely reporting that, say, 90% of a cohort passed an examination, quality of an implemented curriculum could be reported in terms of the percentage distribution at the various SOLO levels.
However, regardless of what a curriculum guide states, especially with regards to SIOs, the teacher makes the final choice as to what is to be presented and what emphasis will be given to the content, materials and activities selected (Oliver, 1977: 94). As such, it is also possible to measure the quality of implemented curriculum between schools and also within schools using the above methodology. The feedback/report culminating from such an evaluation would specifically pin-point the much needed information for classroom teachers and principal in enhancing the quality of teaching and learning. Hence at the school level, more meaning could be derived from this evaluation.
As this evaluation and analysis via the Rasch model is objective, the effectiveness and efficiency of curriculum implementation could also be identified. In the IEA studies, Rosier & Keeves (1991: 299-300) proposed three test validity indexes, namely the Test Coverage Index, Test Relevance Index and Curriculum Coverage Index. Along similar lines, the effectiveness index of a curriculum and efficiency index associated with the curriculum implemented could help quantify the much needed depth of quality curriculum evaluation.
The curriculum effectiveness index (CEI) measures the degree to which students accomplish higher-order thinking (with reference to the SOLO levels) after being exposed to a specific curriculum (contents). Table 6 illustrates how the CEI could be calculated.
Item No:Topic/SIO No:SOLO Levels 1 5.2 1.3 (Alternative at Relational Level) 1.2 (Alternative at Multistructural Level) 1.1 (Alternative at Unistructural Level)
2 10.1 2.3 (Alternative at Relational Level)
2.2 (Alternative at Multistructural Level) 2.1 (Alternative at Unistructural Level) . . . . . .
Student% correct (Level 3)% correct (Level 2)% correct (Level 1) A100 0 0 B 8020 0 C 6040 0 D 402040 E 204040
F 04040 ---------- -------- ---------Total300 160 120
Table 6. Calculation of CEI using hypothetical dataThe CEI for the various levels would help indicate the 'operational levels' of the cohort in question. In the hypothetical example given above, 50% of the cohort who sat for the examination have been effective in operating at the 'Relational Level'. Thus, the CEI at the Relational Level is 50%. Similarly, the CEI at the Multistructural Level is (50% + 27%) 77% [the assumption here is based on Biggs & Collis (1982) - that a student operating at the relational level would be also operational at the multistructural level]. Finally, the CEI at the Unistructural Level is 97%. This index would be particularly useful for both general curriculum evaluation and for within and between school-based curriculum evaluation. Furthermore, comparison between subjects can be undertaken to elicit the various 'operational levels', assuming all other factors stay constant. Such an index would enable appropriate instructional programmes to be effectively placed and tested in various school settings. Thus, apart from reporting general grade-level performance (which could be based on norms), such an 'operational-level' reporting would enable the educators concerned to take appropriate enrichment or remedial actions.
Apart from the CEI, the Implemented Curriculum Efficiency Index (ICEI) would be useful in measuring the efficiency of an implemented curriculum with respect to the time-on-task. Here, time-on-task refers to the curricula time taken to teach a particular topic, which includes the target objectives being tested. The 'time' begin from the introduction of the topic till it is formally concluded (when formative assessment is administered) or a new topic began.
With reference to the above example, say items 1 through 6 cover the topic in general physics. A breakdown is done as above, i.e. arranging the students based on their response level to the six items. The time expected to complete the topic (Planned time) is recorded (from the curriculum specialist/planner). The curriculum time taken (Actual time) to teach this topic is also captured (from the teacher). The efficiency index can be calculated using the relationship:
ICEI = (CEI) x (Planned time) / (Actual time)[4]
Let say it takes 14 hours (actual time) to teach the topic in general physics, and the curriculum planners have estimated that it would take about 12 hours (planned time). Therefore, the ICEI (Level 3) = (50% x 12 / 14) = 43.0%; ICEI (Level 2) = (77% x 12 / 14) = 66.0%; and ICEI (Level 1) = (97% x 12 / 14) = 83.0%. Hence, it is evident from this illustration that the 'teaching program' in a school, is 83% efficient at the Unistructural Level, 66% efficient at the Multistructural Level and 43% efficient at the Relational Level in
implementing the curriculum. It is possible with this ICEI to reach beyond the 100% efficiency and could reflect the rigours of some school-based programs. It must be noted that such quality comparison can be attempted between and within schools. At the national level, trends over the years can be effectively and meaningfully compared and appropriate actions taken accordingly.
Conclusion
This article has attempted to show that a modification to a curricula area is needed is, in part, the consequence of having judged that a deficit exists or that changes holds promise for significant educational gains and thus in quality upgrading. Although based on Piagetian stage theory, the SOLO taxonomy makes a clear distinction between the generalised cognitive structure of the individual, which is a hypothetical concept not directly measurable, and the structure of the actual responses that the students makes to specific learning tasks. This makes the SOLO taxonomy a valuable evaluative and instructional tool (Courtney, 1986).
While SOLO is not in itself a normative or prescriptive theory, it can provide sets of descriptions that may be used prescriptively within a curriculum theory and thus in curriculum evaluation (Biggs & Collis, 1989).
It can be fairly argued that this could mean revising the behavioural objectives in a syllabus to include more higher-order thinking skills-type performance indicators as and when required and not bound by the rigid rule of percentage of knowledge-based and application-based quota, as specified in general Table-of-Specifications (TOS). Such a methodology would imply re-looking into both formative and summative assessments, especially when both pedagogy and curriculum are being 'revamped' by the 'tremors' of information technology (Anderson & Alagumalai, 1996). This gives the impetus to teachers to look beyond their textbooks and model examination papers and force their students to think at a higher level. It is subsumed that students will have a basis in the lower-levels of the SOLO taxonomy before attempting a higher-level response.
The 'coupling' of both the SOLO taxonomy and the RASCH model, enables specific measurement of latent traits and also in predicting models in psychometry and cognitive science. This article has attempted to outline how the quality of a curriculum could be estimated thus enabling effective curriculum evaluation. It has also laid down the basis for objective reporting of effectiveness and efficiency in curriculum implementation, which is an outcome of curriculum eveluation. Thus, the SOLO taxonomy used effectively with the Rasch model for measuring the quality of learning and in promoting higher-order thinking within a curriculum.
In summary, no improvement in educational program can be lasting without a corresponding improvement in the system of evaluation. Thus, it would be a moral and ethical undertaking for curriculum specialist and educational policy makers to evaluate and report objectively curriculum changes and 'shifts' and generally on quality of education for those under them (Kemmis & Stake, 1988: 11).
To conclude, "just as the navigator could not sail a ship without a rudder, so is the educator adrift who has an educational program without a philosophy or sound evaluation in place" (Oliver, 1977: 91).
References
Adams, R.J. & Khoo, S.T. (1994). QUEST: An Interactive Test Analysis System. Melbourne, ACER.
Anderson, J., & Alagumalai, S. (1996). From text-based pedagogy to cyber-based methodology. Paper presented at the ERA-AARE Joint Conference, Singapore. (25-29 November 1996).
Andrich, D. (1988). Rasch Models for Measurement. Newberyy Park, CA:
SAGE.
Archbald, D.A., & Newmann, F.M. (1988). Beyond Standardized Testing. Reston, VA: National Association of Secondary School Principals.
Biggs, J.B., & Collis, K.F. (1982). Evaluating the Quality of Learning. The SOLO Taxonomy. New York: Academic Press.
Biggs, J.B., & Collis, K.F. (1989). Towards a model of school-based curriculum development and assessment: Using the SOLO taxonomy. Australian Journal of Education, 33, 149-161.
Choppin, B. (1983). The Rasch model for item analysis. Center for the study of Evaluation Report No. 219. LA: University of California.
Collis, K.F., & Romberg, T.A. (1981). Construction of super-items for the assessment of mathematical problem solving. Paper presented at the 1981 Australian Association for Research in Educational Annual Conference. Adelaide, Australia.
Courtney, T.D. (1986). The significance of the SOLO taxonomy for learning anf teaching in geography. Geographical Education, 5(2), 47-50.
Cureton, E.E. (1965). Reliability and validity: Basic assumptions and experimental designs. Educational and Psychological Measurement, 25, 327-346.
Dennis, L., Prescott, W., Gammage, P., Greenwald, J., & McMahon, H. (1976). The Child, The School and Society. UK: The Open University.
Eisner, E.W. (1994). The Educational Imagination: On the design and evaluation of school programs. (3rd Ed.). New York: MacMillan College Publishing Company.
Hambleton, R.K., et al. (1978). Developments in latent trait theory: Models, technical issues, and applications. Review of Educational Research, 48, 467-510.
Hambleton, R.K. (1982). Item response theory: The three-parameter logistic model. Center for the Study of Evaluation Report No. 220. LA: University of California
Hass, G. (1983). Curriculum Planning: A New Approach. (4th Ed.) London: Allyn & Bacon, Inc. Eraut, M.R. (1989). Specifying and Using Objectives. Cited in Eraut, M. (Ed.) The International Encyclopaedia of Educational Technology. New York: Pergamon Press.
HusÈn, T. (ed.) (1976). International Study of Achievement in Mathematics. Stockholm, Almquist and Wiksell, and New York: Wiley (2 volumes).
Keeves. J.P. & Alagumalai, S. (1996). Advances in measurement in science education, in Fraser, B. (Ed.) International Science Education Handbook: Volume 3 Kluwer Press. [in print]
Kemmis, S. & Stake, R. (1988). Evaluating Curriculum. Deakin University, Australia.
Kemp, J.E. (1985). The Instructional Design Process. New York: Harper & Row, Publishers.Keeves, J.P. (1968). Variation in Mathematics Education in Australia. Hawthorn, Victoria: ACER.
Keeves, J.P. & Bourke, S.F. (1976). Australian Studies in School Performance. Volume 1: Literacy and Numeracy in Australian Schools. Canberra: AGPS.
Keeves, J.P. & Kotte, D. (1995). The measurement and reporting of key competencies. Flinders Institute for the Study of Teaching Research Summary. Adelaide: Flinders University of South Australia.
Lewis, G.B. (1994). Tertiary students' knowledge of their own learning and a SOLO taxonomy. Higher Education, 28, 387-402.
Lord, F.M. (1980). Applications of item response theory to practical testing problem. Hillsdale, NJ: Erlbaum.
McCormick & James, M. (1983). Curriculum Evaluation in Schools. (2nd Ed.) London: Croom Helm
McCormick & James, M. (1988). Curriculum Evaluation in Schools. London: Croom Helm.
Mislevy, R.J., Yamamoto, K., & Anacker, S. (1991). Toward a test theory for assessing student understanding. Princeton, NJ: Educational Testing Service.
Murnane, R.J. & Raizen, S.A. (1988). Improving indicators of the quality of science and mathematics education in Grades K-12. National Research Council. Washington, D.C.: National Academy Press.
Oliver, A.I. (1977). (2nd Ed.) Curriculum Improvement: A guide to problems, principles and process. New York: Harper & Row, Publishers.
Raizen, S.A. & Jones, L.V. (eds.) (1985). Indicators of Precollege
education in science and mathematics: A Preliminary Review. National Research Council. Washington, D.C.: National Academy Press.
Ramsden, P. (1988). Studying learning: Improving teaching. Cited in Ramsden, P. (ed). Improving Learning: New Perspectives. London: Kogan Page.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Denmark Paedogogistic Institut.
Rasch, G. (1966). An item analysis which takes differences into account. British Journal of Mathematical and Statistical Psychology, 19(1), 49-57.
Rosier, M.J., & Keeves, J.P. (1991). The IEA study of science I: Science education and curricula in twenty-three countries. Oxford: Pergamon Press.
Shane, H.G., & Tabler, M.B. (1983). Complexities of curriculum planning of tomorrow's world. Cited in Hass, G. (1983). Curriculum Planning: A New Approach. (4th Ed.) London: Allyn & Bacon, Inc.
Snyder, S., & Sheehnan, R. (1992). The Rasch measurement model: An introduction. Journal of Early Intervention, 16(1), 87-95.
Wilson, M. (1985). Scaling SOLO science items. Hawthorn, Victoria: ACER.
Wilson, M. (1989). A comparison of deterministic and probabilistic approaches to measuring learning structure. Australian Journal of Education, 33(2), 127-140.
Wright, B.D. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14, 97-116.
Wright, B.D. & Stone, W.H. (1979). Best Test Design. Chicago, MESA Press.
5SOLO, RASCH, QUEST and Curriculum Evaluation Page
