TFR Guide Assessment Glossary 2009-08-01 TVT

Assessment Glossary

The Follow-Up Report: Guides and Handbooks

1

The Follow-Up Report

Assessment, Research, and Planning Glossary (A Guides and Handbooks Report) Compiled by Todd V. Titterud Revised 08/01/2009

Assessment Glossary


2

Assessment, Research, and Planning Glossary

(Compiled by Todd V. Titterud, 05/21/2007, Revised: 08/01/2009)

There are numerous glossaries available online covering technical and common terms prevalent in the fields of assessment, evaluation, planning, administration, research design, statistical analysis, and testing. This glossary was compiled from materials originally downloaded for personal use from many different sources over several years and citations for the original sources are not available. While the glossary is neither definitive nor all inclusive, it does combine terms and their usage from all of these interrelated fields into a single reference. Much of the terminology in these fields is fluid and evolving rather than standardized. Many terms have alternate definitions and there are multiple terms with similar or equivalent meanings and usage. It is often recommended that institutions establishing their own cyclical regime of institutional research, assessment, and planning develop their own glossary of adopted definitions for key terms in the process.

| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R

| S | T | U | V | W | X | Y | Z |

A A-B-A design A single-case experimental design in which the response to the experimental treatment condition is compared to baseline responses taken before and after administering the treatment condition A-B-A-B design An A-B-A design that is extended to include the reintroduction of the treatment condition A priori codes Codes that were developed before examining the current data AA or AS degree requirements The areas in which students must demonstrate achievement in learning to earn an AA or AS Degree; generally demonstrated through successful completion of required and elective courses Ability Present or potential capacity of an individual to perform a task or to use skills, including ones that are intellectual and physical Ability or trait parameter In item response theory, a theoretical value indicating the level of a participant on the ability or trait measured by the test; Analogous to the concept of true score in classical test theory

Assessment Glossary


3

Ability testing The use of standardized tests to evaluate the current performance of a person in some defined domain of cognitive, psychomotor, or physical functioning Absolute decision Decision based on the absolute level of performance in reference to a criterion; A performance decision which is not norm referenced Abstract A brief summary of what is in an article Abuse The use of an existing authority for purposes that extend beyond or contradict the intentions of the grantors of that authority; The furnishing of excessive services to beneficiaries of government programs, violating program regulations, or performing improper practices which do not involve prosecutable fraud Academic aptitude test Test which predicts achievement in academic pursuits; Design needs to minimize the effect of exposure to specific materials or courses of study on the examinee's score Academic program review The periodic peer evaluation of the effectiveness of an educational degree program; usually encompassing student learning, faculty research, scholarship, and service, and assessment resources Access The ability to gain the attention and to influence the actions of key decision makers Accessible population The research participants who are available for participation in the research ACCJC Accrediting Commission for Community and Junior Colleges Accommodations or adaptations Modifications in the way assessments are designed or administered so that students with disabilities or limited English proficiencies can be included in the assessment; A reasonable modification in an assessment instrument or its administration made to compensate for the effects of a qualified disability without altering the purpose of the assessment instrument; For example: Braille forms or tests in native languages Accountability Responsibility for implementing a process or procedure, for justifying decisions made, and for results or outcomes produced; Use of results for program continuance or discontinuance; Public reporting of student, program, or institutional data to justify decisions or policies; Using results for determining funding; Demand by community stakeholders (public officials, employers, taxpayers, donors, investors, etc.) for educational institutions to prove that the money invested in education has led to measurable learning; The responsibility placed on an individual or group for their own or others actions, conduct, performance, projects, etc.; The political principle that agencies or organizations are subject to some form of external control, causing them to give a general accounting of and for their actions; The demonstration or proof to external constituencies that the programs, services, and management of an institution are responsible and effective Accounting The process of classifying, measuring, and interpreting financial transactions

Assessment Glossary


4

Accreditation A voluntary process involving an association of schools and/or colleges to encourage high standards of education; Accreditation indicates that the Commission judges that the institution, in a manner consistent with Commission standards, offers its students on a satisfactory level the educational opportunities implied in its objectives and is likely to continue to do so; A status awarded by a certification agency to a candidate that has demonstrated compliance with the standards set forth in the certification program; Certification awarded by an external, recognized organization, that the institution or program meets certain requirements overall, or in a particular discipline Accrediting body A voluntary, non-governmental association established to administer accrediting procedures; A listed accrediting body is one that is officially listed by the Secretary of Education because it is used as part of the Department of Education’s processes for determining institutional eligibility for certain federal funds Accrediting or Accreditation Liaison Officer (ALO) The individual in an institution assigned to carry on continuing relations with an accrediting body Acculturation The process of adapting to a new culture Accuracy Used in survey research to refer to the match between the target population and the sample; Degree to which the data and information collected about the performance of a student are correct measures of performance and are free from error; Extent to which an evaluation conveys technically adequate information about the performance and qualifications of a student; Extent to which an evaluation is truthful or valid in what it says about a program, project, or material Achievement Ability to demonstrate accomplishment of some outcome for which learning experiences were designed; What a student has learned as a result of formal instruction Achievement or proficiency levels Descriptions of student or adult competency in a particular subject area, usually defined as ordered categories on a continuum, often labeled from "basic" to "advanced," that constitute broad ranges for classifying performance Achievement test or testing Assessment method designed to measure student competency or acquired knowledge, skills, attitude, or behavior in relation to specified learner expectations; An assessment that measures a student's acquired knowledge and skills in one or more common content areas (for example, reading, mathematics, or language); A standardized test designed to efficiently measure the amount of knowledge and/or skill a person has acquired, usually as a result of classroom instruction; May be used to produce a statistical profile used as a measurement to evaluate student learning in comparison with a standard or norm; A test to evaluate the extent of knowledge or skill attained by a test taker in a content domain in which the test taker had received instruction; An objective examination that measures educationally relevant skills or knowledge about such subjects as reading, spelling, or mathematics Acquiescence bias Bias created when respondents in a survey answer positively to two equally conflicting questions Acquiescence response set The tendency to either agree or to disagree Action learning

Assessment Glossary


5

Involves a group of people coming together regularly to work collectively on problems in their community, workplace or project in order to achieve improvement by helping each other learn from their experiences; People involved in action learning increase their skills and knowledge by learning with and from each other, working on real issues or problems, reflecting on their experiences, and taking action based on group decisions; Values associated with action learning include respect for others, honesty and integrity, collaboration, and developing trusting relationships with others Action plan A detailed description of the strategies and steps used to implement a strategic plan; Tactics describing who, what, when, where and how activities will take place to implement a strategy Action research A multi-stage type of research in which a problem is researched, changes are made, the problem is researched again, more changes are made, and so on through a number of cycles, until the problem is solved; Applied research focused on solving practitioner’s local problems; Collecting data in the field to allow for continuous adjustments and improvements; Targeted research done to facilitate immediate decision-making; Goal oriented research done to support effective change; Research that allows the researcher to take an active role as an involved participant in the process of planning and implementing changes; School and classroom-based studies initiated and conducted by teachers and other school staff in an effort to improve classroom instruction and educational effectiveness; Research addressing both the practical concerns of people (in a community, organization, etc.) and the goals of research through people working together on projects; A flexible method of integrating research into projects, involving community participants and generating action which is essential for good planning and development; Effective action research depends on the agreement and commitment of participants; Useful for working on complex social problems or issues that need systematic planning and analysis; Involves an ongoing cycle of planning, acting, observing, reflecting (and back to planning) involving critical reflection as an important step in each cycle; Tends to be cyclic (similar steps tend to recur in a similar sequence so that actions are responsive to emergent learning), participative (clients and informants are involved as partners or active participants in the research process), qualitative (deals more often with language than with numbers), and reflective (critical reflection on the process and outcomes are important parts of each cycle); Inquiry-based research conducted by practitioners for the purpose of making change; Practitioners examine their existing practices, form a question about their practice, collect and analyze or interpret data, and use the research results to enhance and improve their professional practice Active consent A process whereby consent is provided by signing a consent form Active inquiry Students actively participate in learning through exploration and research; Teachers using active inquiry in their classrooms guide and support students to ask and answer their own questions related to what is being studied Activity-based costing Processes or activities which use resource inputs to produce outputs; One of the operating expenses of a university which must be considered in managerial/funding decisions Ad hoc research Research conducted in response to a specific, one-time-only need ADA Americans with Disabilities Act Adaptive testing A sequential form of individual testing in which successive items in the test are chosen based primarily on the psychometric properties and content of the items and the participant's response to previous items

Assessment Glossary


6

Additive and interactive effects Refers to the fact that the threats to internal validity can combine to produce an additive or multiplicative bias Adjusted validity or reliability coefficient A validity or reliability coefficient, usually a product-moment correlation, which has been adjusted to offset the effects of differences in score variability resulting from different populations Administration The management and direction of the affairs of an organization; A collective term for all policy making officials or executive management of an organization Administrative agency A government organization set up to implement a law; Any civilian government body (board, bureau, department, or individual), other than a court or legislature, that deals with the rights of private parties by adjudication, rule making, investigation, prosecuting, etc. Administrative feasibility Extent to which appropriate data are readily available or can be obtained, produced, or interpreted with available resources such as staff expertise, time, and equipment Administrative order A directive carrying the force of law issued by an administrative agency after adjudication Administrative outcomes Operational and specific statements derived from a unit’s core functions that describe the desired quality of key services within an administrative unit and define exactly what the services should promote Administrative responsibility Accountability for the conduct of the evaluation process and for the decisions made based on the evaluation results Administrator Anyone with fiduciary responsibility; A manager; The head of a government agency Advanced placement exams Test and assessments given at the beginning of an instructional segment to determine to what extent students have already achieved the objectives of the planned instruction, so that they may be place in more advanced course work Adversarial groups or teams Group of people who enter into cross-examination of counter plans, strategies, or outcomes Advisory committee A group of individuals appointed or elected to recommend and implement policy related to program operation; A group or panel of internal or external members with no decision making authority, assembled to identify and discuss specific issues and make recommendations Advocacy groups or teams Groups of people who are brought together to develop competing strategies for achieving a given set of objectives Affective

Assessment Glossary


7

Any category of feeling, as distinct from cognition or behavior; The display of affect is a set of physical changes which indicates an emotional state; Outcomes of education involving feelings more than understanding (e.g. likes, pleasures ideals, dislikes, annoyances, and values) Affective dimension The psychological concept that refers to a person's feelings, emotions, or degree of acceptance or rejection of some object Affective domain The range of feelings and emotions: Includes interests, attitudes, motivations, values, and appreciations Affective outcomes Outcomes of education involving feelings more than understanding; For example: Likes, pleasures, ideals, dislikes, annoyances, and values Affinity group A type of focus group in which respondents already know one another Affirmative action Any program, policy or procedure implemented to correct past discrimination and prevent current and future discrimination within an organization or system Affirmative action plan An organization's written plan to remedy past discrimination against, or underutilization of, women and minorities; Usually consists of a statement of goals, timetables for achieving them, and specific program efforts Age equivalent The chronological age in a defined population for which a given score is the median (middle) score Age norms Values representing typical or average performance of members of age groups Agency Any department, office, commission, authority, administration, board, government owned corporation, or other independent establishment of any branch of government in the United States; A formal relation whereby one person is authorized to act for another Agenda setting The process of deciding what issues will be considered at a formal meeting; The process by which ideas or issues come up through the various political or administrative processes to wind up on the agenda of a political institution or policy making body Aggregate A group of persons having certain traits or characteristics in common without necessarily having direct social connection with one another (e.g. all female physicians or all cities with a population over 20,000) Aggregated scores The combined scores for a population of students; Often expressed as an average; Requires that all the scores be based on the same or equivalent assessments administered in uniform ways Aggregating Combining two or more related scores into one total score

Assessment Glossary


8

Aims The broad, long-term goals of a project Alignment The process of assuring that learning outcomes, local curriculum and instruction and the system of assessment all support and match each other; How ideas, goals, outcomes, etc. compare or relate across different areas, levels, courses, etc.; Process of linking content and performance standards to assessment, instruction, and learning in classrooms; Process of strengthening the linkage between job responsibilities and teacher evaluation systems, or between curriculum/instruction and assessment; The process of bringing various elements together to move in the same direction Allocational effects The ways in which policies influence of the use of resources Alternate forms Two forms of the same test, both of which measure the same construct; Two or more versions of a test that are considered interchangeable, in that they measure the same constructs in the same ways, are intended for the same purposes, and are administered using the same directions; A generic term used to refer to any of three categories: Parallel forms: Tests with equal raw score means, equal standard deviations, equal error structures, and equal correlations with other measures for any given population, Equivalent forms: Tests without the statistical similarity of parallel forms, but the dissimilarities in raw score statistics are compensated for in the conversions to derived scores or in form-specific norm tables, or Comparable forms: Tests that are highly similar in content, but the degree of statistical similarity has not been demonstrated Alternative assessment Assessment measures other than traditional, standardized, norm- or criterion-referenced paper and pencil testing; Nontraditional means of recording evidence of learning; Requires students to actively accomplish complex and significant tasks, while bringing to bear prior knowledge, recent learning, and relevant skills to solve realistic or authentic problems; Entails direct observation of student performance; For example: Coding live art criticism discussions, portfolio reviews, rating performances or art products on criteria established by teachers and students, journals, and authentic task assessment; See also authentic and performance assessment Alternative hypothesis Statement that the population parameter is some value other than the value stated by the null hypothesis Ambiguous temporal precedence The inability to specify which variable is the cause and which is the effect Amount technique Manipulating the independent variable by giving the various comparison groups different amounts of the independent variable Analysis Assessments performed by an organization to provide a basis for decision making; Organizing and reviewing information so it has meaning; Conclusions and hypothesis derived from the review of collected and organized data or information; The interpretation of data; The treatment of data and information in order to elicit certain statistical data, assessment results, and evaluative conclusions Analysis of covariance A statistical method that can be used to statistically equate groups that differ on a pretest or some other variable; Also used to examine the relationship between one categorical independent variable and one quantitative dependent variable controlling for one or more extraneous variables

Assessment Glossary


9

Analysis of variance A statistical method used to determine whether a relationship exists among two or more variables by formulating concurrent comparisons of the variables Analytic rubric An analytic rubric articulates levels of performance for each criterion so the teacher can assess student performance on each criterion Analytic scoring A method of scoring in which each critical dimension of performance is judged and scored separately, and the resultant values are combined for an overall score; A type of rubric scoring that separates the whole into categories of criteria that are examined one at a time; Evaluating student work across multiple dimensions of performance rather than from an overall impression; A type of rubric scoring that separates the whole into categories of criteria that are examined one at a time; Products or performances are evaluated for selected dimensions, with each dimension receiving a separate score; For example: A piece of writing may be evaluated on several categories, such as organization, use of details, attention to audience, and language usage/mechanics; Analytic scores may be weighed and totaled; Approach to scoring or rating that considers various parts or aspects of the attribute or performance being assessed, for use in profiling strengths and weaknesses or in obtaining an overall summary; Scores may be recorded as a check mark for presence or absence of an attribute, marked on a numerical or descriptive rating scale, or put in the form of a brief comment; The use of a scoring key (e.g. rubric) containing an ideal response to judge the competence or proficiency of student responses on an assessment; The use of specific criteria or features to evaluate and assign points to each essential part of a product or performance Analytical skills The ability to discover the underlying structure of an argument, a communication, a problem, or a solution Analytical thinking The ability to analyze facts, generate a comparison, and draw correct inferences or conclusions from available information Anchors A written descriptor for a point on a rating scale; Descriptive point on a scale or continuum; Representative products or performances used to illustrate each point on a scoring scale; Actual samples of student work which illustrate the essential characteristics of work typical for each scoring level on a scoring rubric; Raters use anchors to score student work, usually comparing the student performance to the anchor; For example: If student work was being scored on a scale of 1-5, there would typically be anchors (previously scored student work) exemplifying each point on the scale; Anchors for the highest score point are sometimes referred to as exemplars Anchor test A common set of items administered with each of two or more different forms of a test for the purpose of equating the scores of these forms ANCOVA (Analysis of Co-Variance) Same method as ANOVA, but analyzes differences between dependent variables Androgogy Adult instructional methods; methods designed to have more appeal or effectiveness with adult learners; The art and science of teaching adults; The art and science of helping adults learn Anecdotal record A short, written report of an individual's behavior in a specific situation or circumstance; a short narrative of an event or activity that may be used to support generalizations about the performance of a teacher

Assessment Glossary


10

Angoff A standard-setting procedure, which requires participants to review test items and to provide estimation of the proportion of a subpopulation of examinees who would answer the item correctly Annotated rubric Notes from an assessment development group, often after a field test and initial scoring, which explain the meaning of criteria or distinctions between the criteria on a rubric; Used to increase scoring reliability and to train others to score consistently Announced visit or observation Student observation or visit that is prearranged with the student to be evaluated and for which the faculty can prepare; Class observation or visit that is prearranged with the instructor to be evaluated and for which the instructor can prepare Annual update or report A brief report, submitted annually, from each academic program based on its assessment plan and which outlines how evidence was used to improve student learning outcomes through curricular and/or other changes or to document that no changes were needed Anonymity A situation in which it is not possible to identity an individual; Evaluator action to ensure that the identity of subjects cannot be ascertained during the course of a study, in study reports, or in any other way; Keeping the identity of the participant from everyone, including the researcher ANOVA (Analysis of Variance) A method of statistical analysis broadly applicable to a number of research designs and used to determine differences among the means of two or more groups on a variable; A simple summary of the variation in the experimental data; One of the most widely used statistical methods; The independent variables in an ANOVA are usually nominal, and the dependent variable is usual an interval; A class of parametric statistical procedures capable of processing more than two columns of group-difference data in one row (one way ANOVA) or two or more columns of such data in two or more rows (two way or factorial ANOVA) Apparency A clear, understandable representation of the data Apparent authority The appearance that an individual has the authority or power to act as an organization’s agent even though the organization has bestowed no such authority or power to that individual Applicability The extent to which information on a particular outcome measure meets the needs of multiple stakeholder groups; For example: The extent to which data generated from a critical thinking, problem solving, or writing assessment yield information that can be used by multiple groups, such as faculty and administrators who wish to improve programs, or government officials and prospective employers who desire documentation of skill level achievement or attainment Application service provider (ASP) Third-party organizations that deliver software applications and related services over the Internet, allowing an organization to outsource some or all of its information technology needs Applied research Research focused on answering practical questions to provide relatively immediate solutions; Research conducted with the intention of applying the results to some specific problem; In applied research the research

Assessment Glossary


11

questions are generally determined by policy makers, decision makers, or other stakeholders rather than by researchers; Systematic study to gain knowledge or understanding necessary to determine how a recognized and specific need may be met Appreciative Inquiry A large-group method, often used at times of organizational change, to discover the most valuable aspects of the organization's past that should be carried forward into its future Appropriation Money that has been set aside and allocated for a specific function Aptitude The potential for acquiring abilities or developing competencies; A student's capability or potential for performing a particular task or skill Aptitude test A test consisting of items selected and standardized so that the test predicts a person's future performance on tasks not obviously similar to those in the test; A test intended to measure the test-taker's innate ability to learn, given before receiving instruction; A test that estimates future performance on other tasks not necessarily having evident similarity to the test tasks; Tests that focus on information acquired through the informal learning that goes on in life; Often aimed at indicating an individual's readiness to learn or to develop proficiency in some particular area if education or training is provided; May not differ in form or substance from achievement tests while differing in use and interpretation Arbitrary A characteristic of a decision or action that is uninformed or is based upon one person's judgment or discretion Archival records Existing biographical, academic, or other file data available from the college or other agencies and institutions Archival search An examination of existing records, reports, and documents pertaining to the object of the evaluation Archived research data Data originally used for research purposes and then stored Arithmetic mean A kind of average obtained by dividing the sum of a set of scores by the total number of scores; Usually referred to as the mean Articulation of quality An expanded articulation of an ideal future condition of quality, arising from the words and phrases of a strategic plan Artifact A product developed by the student or another individual; An artificial statistical phenomenon or result; Assessment or statistical findings which result from errors or weaknesses in the research design or testing instrument; Something that is artificial, contrived, or fictitious; A research conclusion which arose due to a bias or confound of some sort, rather than because of the causal relationship being tested Assessment The collection and analysis of evidence of effectiveness for all parts of the institution, including but not limited to evidence of student learning and achievement; Process of gathering, describing, or quantifying information about performance; The process of observing learning by describing, collecting, recording, scoring, and

Assessment Glossary


12

interpreting information about a student's or one's own learning; Process of collecting information about a student to aid in decision making about the progress and development of the student; The process of or instrument for measuring, quantifying, and/or describing those aspects of teaching related to the attributes covered by the evaluation; Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development; An effort to gather, analyze and interpret evidence that describes institutional, departmental, divisional, or program effectiveness; Ongoing process to assure and document learning; Ongoing process aimed at understanding and improving student learning; To thoroughly and methodically analyze student accomplishment against specific goals and criteria; The systematic collection and analysis of information to improve student learning; Assessment is not an end in itself nor is it merely to gather information, rather the purpose of assessment is educational or learning improvement; Involves making expectations explicit and public, setting appropriate criteria and high standards for learning quality, systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards, and using the resulting information to document, explain, and improve performance; Individual student accomplishments are assessed and summarized to ascertain the achievement of groups or subgroups of students Assessment or evaluation audit An independent examination and verification of the quality of an evaluation plan, the adequacy of its implemented, the accuracy of results, and the validity of conclusions; An independent quality check and verification of the assessment and evaluation of a teacher Assessment for accountability Assessment of some unit (e.g. program, department, college, or entire institution) to satisfy stakeholders external to the unit itself; Results are summative and are often compared across units Assessment for improvement Assessment that feeds directly, and often immediately, back into revising the course, program or institution to improve student learning results; May be formative or summative Assessment instrument The evidence of learning that the student will produce (e.g. test, essay, portfolio, demonstration) which will be evaluated by faculty with respect to a rubric Assessment literacy The possession of knowledge about the basic principles of sound assessment practice; Includes knowledge about terminology, the development and use of assessment methodologies and techniques, familiarity with standards of quality in assessment, and familiarity with alternatives to traditional measurements of learning Assessment method A strategy or technique evaluators may use to acquire evaluation information; For example: Observations, text- and curriculum-embedded questions and tests, paper-and-pencil tests, oral questioning, benchmarks or reference sets, interviews, peer- and self-assessments, standardized criterion-referenced and norm-referenced tests, performance assessments, writing samples, exhibits, portfolio assessment, and project or product assessments Assessment of individuals Uses the individual student, and his/her learning, as the level of analysis; May be quantitative or qualitative, formative or summative, standards-based or value added, and used for improvement; Needs to be aggregated if used for accountability purposes; For example: Improvement in student knowledge of a subject during a single course, or improved ability of a student to build cogent arguments over the course of an undergraduate career Assessment of institutions Uses the institution as the level of analysis; May be quantitative or qualitative, formative or summative, standards-based or value added, and used for improvement or for accountability; Ideally institution-wide goals

Assessment Glossary


13

and objectives serve as a basis for the assessment; For example: How well students across the institution can work in multi-cultural teams as sophomores and seniors Assessment of programs Uses the department or program as the level of analysis; May be quantitative or qualitative, formative or summative, standards-based or value added, and used for improvement or for accountability; Ideally program goals and objectives serve as a basis for the assessment; For example: How sophisticated a close reading of texts senior English majors can accomplish (if used to determine value added, results would be compared to the ability of newly declared majors) Assessment plan A plan written in the first step of the assessment cycle in which faculty draft an SLO for a course or program, outline how it will be assessed, and state how the resulting data will be used to improve student learning: A document that outlines the student learning outcomes (for academic programs) or unit outcomes (for support units), the direct and indirect assessment methods used to demonstrate the attainment of each outcome, a brief explanation of the assessment methods, an indication of which outcome(s) is/are addressed by each method, the intervals at which evidence is collected and reviewed, and the individual(s) responsible for the collection/review of evidence; The written assessment document with the mission, goals, and objectives of the institution, the outcomes to be assessed, and the assessment strategies; A document which identifies expected outcomes for an educational program and outlines how and when the identified outcomes will be assessed Assessment report or record Annual report of the previous year’s assessment activities; The summary of the results and uses of results from the prior year’s assessments; An annual document, based on the assessment plan, that presents and explains assessment results and shows how assessment results are being used to improve the educational program Assessment results The differences or similarities between intended and actual outcomes of the assessment Assessment system Combination of multiple assessments into a comprehensive reporting format that produces comprehensive, credible, dependable information upon which important decisions can be made about students, schools, districts, or states; May include norm-referenced or criterion-referenced assessment, alternative assessment, and classroom assessments Assessment task An illustrative task or performance opportunity that closely targets defined instructional aims, allowing students to demonstrate their progress and capabilities Assessment validity The degree to which an assessment measures what is intended, as opposed to what is not intended, or what is unsystematic or unstable Assimilation The process of being fully absorbed and incorporated into a culture Assumptions Conclusions drawn without data to support them; Something consciously or unconsciously believed to be true without reference to supporting evidence; Recognized or unrecognized beliefs which influence cognitive, affective, and observable behavior; Explicit assumptions are unproven or unprovable beliefs or conditions stated by a researcher to have influenced an analysis or research design; Implicit assumptions are beliefs or conditions inherent or evident in an analysis or research design which are not explicitly stated by the researcher; Implicit assumptions not recognized by the researcher may result in researcher bias; Implicit assumptions not recognized by the research audience or consumer, or the unconscious assumptions of the research audience,

Assessment Glossary


14

may result in the misinterpretation of research results and actions or decisions not supported by the research findings Attainment To gain or accomplish credit or a credential, such as a certificate or degree or educational level Attention assessment The process of collecting data and making an appraisal of a person's ability to focus on the relevant stimuli in a situation; May be directed at mechanisms involved in arousal, sustained attention, selective attention and vigilance, or limitation in the capacity to attend to incoming information Attention bias A type of bias where the act of observation itself becomes an important independent variable or treatment (e.g. the Hawthorne effect) Attribute A characteristic, capacity, or perceived quality of an individual, thing, or place; Individual attributes include, but are not limited to, attitude, ability, behavior, skill, knowledge, or interest Attrition Loss of students through means other than graduation; Loss of subjects from the defined sample during the course of a research study Audience Those persons to be guided by the results of student evaluations in making decisions about the development and progress of students and all others with an interest in the evaluation results and findings; Those persons who will be guided by the evaluation in making decisions and all others who have a stake in the evaluation; Those individuals who have a potential interest in the results of teacher performance assessment and evaluation and in the quality of teaching Audit A systematic assessment of a organization’s management, finances, operations, controls and scope in which policies and procedures are carried out; Inspection of the accounting records and procedures of a business, government unit, or other reporting entity by a trained accountant for the purpose of verifying the accuracy and completeness of the records Audit review approach Form of evaluation which involves measuring how well project objectives were met and the impacts of projects, focusing on questions such as “what did we set out to achieve?” and “what are the signs that we have done this?”, and using methods such as structured surveys Audit trail A record depicting all the chronological stages of a particular transaction Authentic A characteristic of assessments that have a high degree of similarity to tasks performed in the real world; The more authentic the assessment, the less inference required to predict student success after graduation Authentic assessment A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills with performance typically scored on a rubric to determine how successfully the student has met specific standards; A form of performance assessment in which the activity simulates the contexts, tasks and ways people in “real-life” situations use knowledge and skills; An authentic task is one in which the student addresses a complex problem situation under real-life conditions; Assessment

Assessment Glossary


15

tasks that evoke demonstrations of knowledge and skills in ways that they are applied in the “real world”; Assessment that fits meaningful, real-life learning experiences; May include recording evidence of the learning process, applications in products and performances, perception of visual and audio relationships, integrations of new knowledge, reflecting profitably on one's own progress, and interpreting meaning in consideration of contextual facts; Method of assessment in which the student is expected to demonstrate his or her competence and proficiency through the completion of a task that mimics a job, higher educational, or life skill Authentic performance assessment An evaluation of a student's ability to perform in the classroom a complex task that is common in the field or “real world”; Substitute for truly authentic real world tasks which occur infrequently in most classrooms Authentic task An assignment given to students designed to assess their ability to apply standards-driven knowledge and skills to real-world challenges; A task is considered authentic when students are asked to construct their own responses rather than to select from ones presented and the task replicates challenges faced in the real world Authority Power defined according to a legal and institutional framework, and vested in a formal structure (a nation, organization, profession); Power exercised through recognized, legitimate channels Average A general term applied to the various measures of central tendency; The three most widely used averages are the mean (or arithmetic mean), the median, and the mode; When the term "average" is used without designation as to type, the general assumption is that it refers to the arithmetic mean Average deviation A statistical term referring to the average of the deviation, taken from some fixed value, usually the mathematical mean of the data Award units The generic term which describes various measures of the amount of value (or credit) received by a student for the successful completion of a course; A measure of the amount of value (or credit) received by a student for the successful completion of one or more educational experiences, such as a course or other type of learning experience; The number of units to be awarded for the successful completion of a course is usually determined by a faculty committee (or equivalent body) upon consideration of factors such as difficulty of the course, the time expected of a student in contact with the faculty and otherwise (i.e. Contact Hours/Clock Hours and Total Hours), and the type of educational experience (e.g. lecture, laboratory, independent study); When data pertaining to Award Units is transmitted outside the institution, the transmitting institution should specify the Award Unit used (e.g. Semester Credit Hour, CEU), and define the unit, where applicable, in terms of the total number of Award Units required for program completion and, if appropriate, the real-time factors of the Award Unit (i.e. Contact Hours/Clock Hours and Total Hours) Awareness of the hypothesis Confounding due to a research participant's understanding of the research purpose, consciously or unconsciously, influencing the behavior being measured Axial coding The second stage in grounded theory data analysis

Assessment Glossary


16

B Backload Amount of effort required after the data collection Backstage behavior What people say and do only with their closest friends Balanced scorecard A popular strategic management concept, the balanced scorecard is a management and measurement system that enables organizations to clarify their vision and strategy and translate them into action; The goal of the balanced scorecard is to tie business performance to organizational strategy by measuring results in four areas: financial performance, customer knowledge, internal business processes, and learning and growth Bar graph A graph that uses vertical bars to represent the data Barriers to entry Factors that reduce entry into an industry or market BARS See behaviorally-anchored rating scale Baseline The behavior of the participant prior to the administration of a treatment condition; Base level of previous or current performance that can be used to set improvement goals and provide a basis for assessing future progress Basic research Research aimed at generating fundamental knowledge and theoretical understanding about basic human and other natural processes; Systematic study and investigation undertaken to discover new knowledge, facts or principles (the pursuit of knowledge for the sake of knowledge) Battery A set of several tests designed to be administered as a unit; A group of several tests standardized on the same sample population so that results on the several tests are comparable; A group of carefully selected tests that are administered to a given population, the results of which are of value individually, in combination, and totally; Sometimes loosely applied to any group of tests administered together, even though not standardized on the same subjects; Most common test batteries are those used to measure school achievement, which include subtests in the separate learning areas Behavior The combined observable responses of a person to internal and external stimuli Behavior management Structure and support for students to develop appropriate behavior by building on student strengths and developing their confidence in their own abilities Behavior modification A conscious attempt to change or eliminate an individual’s undesirable behavior by specifying expected behavior and reinforcing and rewarding desired behavior

Assessment Glossary


17

Behavior summary scale (BSS) Type of rating assessment in which judgments about teacher performance are related to general or abstract benchmarks representing various levels of performance Behavioral observation scale (BOS) Type of rating assessment in which judgments about teacher performance are related to a series of statements describing specific examples of observable teacher behaviors Behavioral observations Measuring the frequency, duration, typology, etc. of student actions, usually in a natural setting with non-interactive methods (e.g. formal or informal observations of a classroom); Observations are most often made by an individual and can be augmented by audio or videotape Behavioral risk management The process of analyzing and identifying workplace behavioral issues and implementing programs, policies or services most suitable for correcting or eliminating various employee behavioral problems Behaviorally-anchored rating scale (BARS) Type of rating assessment in which judgments about teacher performance are empirically linked to specific examples of incumbent performance at each level of effectiveness on the rating scale Behaviors Actions which are specific and observable; Specific, observable actions of a student in response to internal and external stimuli Bell curve or bell-shaped curve A frequency distribution graph of statistics; A normal distribution shaped like a bell; The curve representing the normal distribution of a rating or test score Benchmark A referenced behavior for comparing observed performance at a given level; The standards used as a basis for comparison or measurement; A detailed description of a specific level of student performance expected of students at particular ages, grades, or development levels; A description or example of student, candidate, or institutional performance that serves as a standard of comparison for evaluation and judging quality; A mathematical average of past performance levels; The first or last set of data used for comparisons; A criterion-referenced objective; Performance data used for comparative purposes; Past performance used as a baseline against which to compare future performance; An exemplary or best practice performance level used as a target to strive for; Exemplary samples of performance or production; An average or standard used for comparisons; A set of benchmarks can be used as "checkpoints" to monitor progress toward meeting performance goals within and across grade levels; An actual measurement of group performance against an established standard at defined points along the path toward the standard with subsequent measurements of performance using the benchmarks to measure progress toward achievement; A criterion-referenced objective; Performance data that are used for comparative purposes Benchmark tasks Pieces of student work selected by a group of lead teachers as exemplifying a certain score level; See also Anchor test Benchmarking An actual measurement of group performance against an established standard at defined points along the path toward the standard; Subsequent measurements of group performance use the benchmarks to measure progress toward achievement; The systematic process of comparing an organization’s products, services and practices against those of a competitor organization or other industry leader to determine what allows it to

Assessment Glossary


18

achieve high levels of performance; Gathering information about model efforts or best practices by other organizations engaged in similar endeavors to help establish project targets and goals; The process of continuously comparing and measuring an organization against recognized leaders and similar organizations to gain information that will help the organization take action to improve its performance; A process of comparing organizational performance against other, similar operations or jurisdictions, a standard, or a past time-frame, while maintaining direct comparability of performance measures Benefit An advantageous consequence of a program or action Benefit-cost ratio The ratio of the present value of benefits over the present value of costs Best practices Practices that enable an organization or individual to achieve superior performance results Bias Systemic variance that skews the accurate reporting of data in favor of a particular group; A constant error; Any systematic influence-on measures or on statistical results-irrelevant to the purpose of the evaluation; A systematic tendency toward a lack of objectivity, fairness, or impartiality on the part of the assessor or evaluator, often based on personal preferences and inclinations; Systematic error in the assessment instrument and procedures, or in the interpretation and evaluation process; Occurs in testing when items systematically measure differently for different ethnic, gender, or age groups; Types of potential bias in research include: attention bias, centripetal bias, confounding, cultural bias, demand characteristics, expectancy bias, measurement bias, recall bias, sampling bias, volunteer bias, and withdrawal bias Biased sample A sample that is systematically different from the population Binomial sign test A single sample inferential statistic for non-parametric data at the nominal or ordinal level; Can also be used as a two-sample inferential statistic for non-parametric data in a related design Bivariate analysis Pertaining to two variables only Blind The giving or taking of measurements without knowledge of the true purpose of the research; See double blind and single blind Bloom's taxonomy Six levels of cognitive ability arranged in order of increasing complexity (from 1=low to 6=high): 1. Knowledge: Recalling or remembering information without necessarily understanding it (includes behaviors such as describing, listing, identifying, and labeling); 2. Comprehension: Understanding learned material (includes behaviors such as explaining, discussing, and interpreting); 3. Application: The ability to put ideas and concepts to work in solving problems (includes behaviors such as demonstrating, showing, and making use of information); 4. Analysis: Breaking down information into its component parts to see interrelationships and ideas (related behaviors include differentiating, comparing, and categorizing); 5. Synthesis: The ability to put parts together to form something original (involves using creativity to compose or design something new); 6. Evaluation: Judging the value of evidence based on definite criteria (behaviors related to evaluation include concluding, criticizing, prioritizing, and recommending) Body of evidence All factual information which bears on the determination reached in a study

Assessment Glossary


19

Body of work (BoW) A standard-setting procedure in which panelists examine complete student response sets and match each student response set to a performance level category based upon previously agreed upon descriptions of what students at the different levels should know and be able to do Bond Promise to repay a certain amount (principal) at a certain time (maturity date) at a particular rate of interest Bonferroni's correction for multiple comparisons An adjustment to the confidence level required when a single scientific hypothesis is being investigated using multiple inferential statistics where the risk is the fact that the probability of a Type 1 error (or p-value) increases with every new statistical procedure Bookmark A procedure used to establish a criterion-referenced cutoff of a score or rating determined by content area experts Boolean operators or connectors Words used to create logical combinations (e.g. “and”, “or”, or “not”) Bootstrapping A non-parametric approach for making statistical inferences employing large numbers of repetitive computations to estimate the shape of a statistic's sampling distribution BOS Behavioral observation scale Box plot or box-and-whisker plot Graphical aid (drawn either horizontally or vertically) designed to display both the central tendency and the dispersal of a given distribution by displaying five values: the lower and upper extremes (the smallest and largest observations, which define the range), the median (the midpoint of the distribution), and the lower and upper quartiles (defined by the 25th and 75th percentiles) Bounded rationality Seeking the best possible solution, but not necessarily the most rational from a purely economic standpoint Bracket To suspend your preconceptions or learned feelings about a phenomenon Branding The process of identifying and differentiating an organization’s products, processes or services from other competing organizations by associating it with a name, phrase or other mark; Process of creating a unique name and image for a product, organization, or individual that attracts and retains loyal customers through advertising or marketing campaigns having a consistent theme Brainstorming Decision-making technique where an individual or group generates multiple ideas and suggestions about a particular topic or theme within a very short time frame; A not very successful method for generating ideas, in which a small group of people come up with ideas as fast as they can and participants may build on each others' ideas, but not criticize them Break-even analysis

Assessment Glossary


20

A measure used to determine the approximate sales volume or income required to cover the costs associated with producing a particular product or service BSS Behavior summary scale Budget A numerical summary of an organization’s available resources and how those resources are to be allocated based on anticipated future expenditures for various items; A plan in numbers Budget padding Proposing a higher budget than is actually needed; Frequently justified by the assumption that all proposed budgets will be reduced prior to authorization regardless of actual needs Bureaucracy A formal organizational arrangement characterized by division of labor, job specialization with no functional overlap, exercise of authority through a vertical hierarchy (chain of command) and a system of internal rules, regulations, and record keeping Business literacy The knowledge and understanding of the financial, accounting, marketing and operational functions of an organization Business planning A plan that determines how a strategic plan will be implemented; Specifies how, when, and where a strategic plan will be put into action; Also known as tactical planning Buy-in Obtaining agreement from key stakeholders that the proposed plan is acceptable

Assessment Glossary


21

C CAAP (College Assessment of Academic Proficiency) A national standardized test by ACT (Academic College Testing) that measures general education proficiencies in Writing, Reading, Math, Critical Thinking and Science Calibration The process of setting the test score scale, including mean, standard deviation, and possibly shape of score distribution, so that scores on a scale have the same relative meaning as scores on a related scale Canned, packaged, or off-the-shelf instruments or software Preexisting tests or software that is designed to be applicable to a variety of situations or applications Capacity Instructor’s potential for acquiring skills and competencies through such means as self-study, on-the-job training, mentoring, coaching, and professional development activities Capacity building The development of an organization’s core skills and capabilities (e.g. building leadership, management, finance and fund-raising, programs, and evaluation) in order to build the organization’s effectiveness and sustainability; The process of assisting an individual or group to identify and address issues and gain the insights, knowledge, and experience needed to solve problems and implement change Capital Assets that are available for on-going business needs to produce income; The fixed non labor factors used in the production of goods and services, and the value of such factors Capital budgeting The separation of expenditures that produce long-term benefits, especially those involving the construction of facilities, from the annual operating costs of an organization; The process for reviewing expenditure decisions for capital projects and deciding on the methods for financing them Capital expenditure A dispersement for the purchase, replacement, renovation or expansion of facilities that cannot be charged as an expense under generally accepted accounting standards Capital expense budget Budgets used to determine capital expenditures for constructing or upgrading facilities and equipment for which a predetermined dollar value has been assigned based on its life expectancy Capital grants Grants for use in construction or renovation Capital improvement Addition to an organization’s fixed assets; A permanent addition to an organization’s fixed assets resulting from an outlay of funds over a specific and finite period of time Capital plan A plan for maintaining assets to continue programs Capricious

Assessment Glossary


22

That which is unpredictable, unaccountable, impulsive, or without a supporting rationale, or which is characterized by implied unruly or whimsical behavior Capstone assessment Assessment of outcomes structured into learning experiences occurring at the end of a program; Capstone experiences involve demonstration of a comprehensive range of program outcomes through some type of product or performance; Capstone outcomes may be those of the degree program only or a combination of those from the program, general education, and the institution as a whole Capstones Capstone experiences for graduating seniors are designed to demonstrate comprehensive learning in the major through some type of product or experience; In addition to emphasizing work related to the major, capstone experiences can require students to demonstrate how well they have mastered important learning objectives from the institution’s general studies program Carryover effect A sequencing effect that occurs when performance in one treatment condition is influenced by participation in a prior treatment condition(s) Case A bounded system Case study A form of qualitative research that is focused on providing a detailed account and analysis of one or more cases; An intensive, detailed description and analysis of a single project, program, or instructional material in the context of its environment; The collection and presentation of detailed information about a particular participant or small group, and frequently includes the accounts of the subjects themselves; A study containing qualitative data (e.g. observations and information drawn from interviews) about one subject; A research method which focuses on the characteristics, circumstances, and complexity of a single case, or a small number of cases, often using multiple methods; Case study findings may raise awareness of general issues, but the aim is not to generalize the findings to other cases; Case studies are conducted primarily using qualitative research techniques, but do not exclude quantitative research methods CAT Classroom Assessment Technique; Also Computer Administered Test Categorical or project grant Grants requiring that the money may be spent for only a limited purpose Categorical variable A variable that varies in type or kind Category midpoint The midpoint between the upper and lower measures in a category range; The midpoint of categories is used to calculate means and standard deviations in cases where the data categories were grouped or collapsed; The arithmetic average of the upper and lower boundaries in a category; For example: The midpoint in the pre grouped category "9 to 11 years of school" is (8.5+11.5)/2 = 10, and for a single score, such as "3," the boundaries would range from 2.5 to 3.5 with an average midpoint of (2.5 + 3.5)/2 = 3 Causal-comparative research A form of non experimental research in which the primary independent variable of interest is a categorical variable Causal description

Assessment Glossary


23

Describing the consequences of manipulating an independent variable Causal explanation Explaining the mechanisms through which and the conditions under which a causal relationship holds Causal line A temporal series of events so related that, given some of them, something can be inferred about the others whatever may be happening elsewhere; A succession of lesser cause-effect events: Event A causes State B, which triggers Event C, etc. Causal model A model which represents a causal relationship between two variables Causal modeling A form of explanatory research where the researcher hypothesizes a causal model and then empirically tests the model; Also called structural equation modeling or theoretical modeling Causal relationship The relationship established showing that an independent variable, and nothing else, causes a change in a dependent variable; Also establishes how much of a change is shown in the dependent variable Causation A situation in which one factor determines the outcome of another factor or result; Fundamental notion of science that some things just happen randomly while others are contingent upon prior events taking place or specified antecedent conditions being reached Causality The relation between cause and effect Cause and effect relationship Relationship in which one variable affects another variable Ceiling Highest limit of performance that can be assessed or measured by an instrument or process; The upper limit of performance or ability that can be measured effectively by a particular test; Individuals have reached the ceiling of a test when they perform at the top of the range in which the test can make reliable discriminations; Individuals who perform near to or above this upper limit are said to have reached the ceiling, and the assessment may not be providing a valid estimate of their performance levels; Such individuals should be given a more difficult assessment of the same attribute if it is necessary to differentiate between higher levels of performance Cell A combination of two or more independent variables in a factorial design Census Survey of a whole population; The entire population of the group in question is included in the study; A study of the whole population rather than a sample Central tendency Measures that indicate the middle or center of a distribution; A measure of central tendency provides a single most typical score as representative of a group of scores; The "trend" of a group of measures as indicated by some type of average, usually the mean or the median Central tendency effect

Assessment Glossary


24

A type of rater effect or artifact in which an assessor or an evaluator tends to rate instructors or students toward the mid-point of a scale or to judge a performance as average or neutral when it is actually well above or well below the middle level of the scale; Consequently assessment results will lack validity because observers lack inter-rater reliability Centrality Core purposes Centralization The process of consolidating all decision-making authority under one central group or location Certification Official recognition of advanced status, outstanding performance or a high level of expertise in the profession; Sometimes used as a synonym for credential or license; A document issued to a person completing a course of study not leading to a degree; Verification that a level of achievement has met designated standards Centrifugal and centripetal bias Types of bias where the research site itself results in samples which are unrepresentative of the population at large; For example: Respondents at low or high prestige institutions may differ from the population as a whole because low prestige sites may risk a centrifugal bias (due to avoidance by respondents of certain characteristics) and high prestige sites may risk a centripetal bias (due to attracting respondents of certain characteristics); Both of these biases illustrate how a random sample can create sampling errors when the experimental sample is drawn from a subset of the parent population created through a self-selection processes CIRP (Cooperative Institutional Research and Effectiveness Program) A national annual survey conducted by the Higher Education Research Institute (HERI) designed specifically for first-time college attendees (includes new transfers into a college); The survey is administered within the first two weeks of school and measures demographic, attitudinal, and educational self-assessment characteristics of an entering cohort Chain of command Structure of decision-making responsibilities from the highest levels of authority to the lowest levels Change verbs Action words that tend to describe how things become different; Useful in writing mission statements, goals, and strategic objectives Changing-criterion design A single-case experimental design in which a participant’s behavior is gradually altered by changing the criterion for success during successive treatment periods Checklist An assessment that is based on the examiner observing an individual or group and indicating whether or not the assessed behavior is demonstrated; List of performance criteria for a particular activity or product on which an observer marks the student's performance on each criterion using a scale that has only two points; For example: Present or absent, and adequate or inadequate; A list of items to be completed for use in the evaluation of constructed response format items; Characteristics or behaviors marked as either present or absent Chi-square or chi-squared test Statistical test used to determine whether a relationship observed in a contingency table is statistically significant; Most common method of statistical analysis for frequency data; Given an array of actual cell frequencies, a null hypothesis expected distribution is computed, and the difference between the two is tested to see if it could have occurred by chance

Assessment Glossary


25

Churn rate The number of participants who discontinue their use of a service over a given period of time divided by the average number of total participants Classical test theory or classical true score theory The view that an individual's observed score on a test is the sum of a true score component for the participant, plus an independent measurement error component; A few simple premises about these components lead to important relationships among validity, reliability, and other test score statistics; A testing theory which postulates error scores are random with a mean of zero, and uncorrelated with error scores on a parallel test and with true scores; Postulates that true scores, observed scores, and error scores are linearly correlated and are represented by the equation x = t + e, where x is the observed score, t is the true score, and e is the error score Classification accuracy The degree to which neither false positive nor false negative categorizations and/or diagnoses, occur when a test is used to classify an individual or event Classroom assessment An assessment developed, administered, and scored by a teacher or set of teachers with the purpose of evaluating individual or classroom student performance on a topic; The systematic and on-going study of what and how students are learning in a particular classroom; Often designed for individual faculty who wish to improve their teaching of a specific course; Differs from tests and other forms of student assessment in that it is aimed at course improvement, rather than at assigning grades; Simple, non-graded, anonymous, in-class activities that help instructors gather feedback from students on the teaching-learning process and determine how well their students are learning what they are being taught; The purpose of classroom assessment is to provide faculty and students with information and insights needed to improve teaching effectiveness and learning quality; College instructors use feedback gleaned through Classroom Assessment to inform adjustments in their teaching; Faculty also share feedback with students, using it to help them improve their learning strategies and study habits in order to become more independent, successful learners; Classroom assessment is a type of formative evaluation Classroom assessment technique (CAT) Assessment techniques embedded into the classroom-based, or other, learning process; Assessment conducted in the classroom because students benefit from opportunities to give and receive feedback on their learning before being evaluated or graded and they also need to learn self-assessment; Faculty benefit from gathering data which address questions about issues or problems in their teaching Classroom management The management of student behavior and learning activities by teachers by, for example, designing learning activities in ways that structure relationships to support learning, the use of rewards and sanctions, and negotiating classroom codes of conduct Classroom research A collaborative process for investigating teaching and learning issues Client Anyone whose interests are served by an organization, or who receives or uses an organization’s resources or services; Clients can be internal to an organization (e.g. one department may be the client of another department) or external to the organization Closed-ended question A question that forces participants to choose a response Closing the loop

Assessment Glossary


26

Using assessment results for program change and improvement; Demonstrating the linkage between assessment and change; Communicating to constituents the changes made due to an assessment; Using assessment results for program change and improvement; The results of one year’s work in strategic planning indicate what the next year’s plan should focus on Cluster A collective type of unit that includes multiple elements Cluster analysis This is one of the four recognized types of multivariate analysis (the others being principal components analysis, factor analysis, and discriminate analysis) Cluster sampling Type of sampling in which clusters are randomly selected; Sampling from selected clusters of potential subjects within the target population, rather than the population as a whole (thus saving time and expense, but at the risk of introducing a sampling bias) Co-curricular programs Out-of-class activities (e.g. student affairs programs and activities) Co-occurring codes Sets of codes that partially or completely overlap Coaching Classroom observation and feedback on specific aspects of practice by colleagues Code A numeric or alphabetic symbol used to represent attributes or words; To translate a given set of data or items into a set of quantitative or qualitative symbols Coding The process of transforming data, evidence, information, judgments, notes, and responses to numeric and/or alphabetic codes; Marking segments of data with symbols, descriptive words, or category names; Making sense of a text by identifying, summarizing, and labeling its components relative to a research question Coefficient A value expressing the degree to which some characteristic or relation is to be found in specified instances; For example: The coefficient of correlation is a value expressing the degree to which two variables vary concomitantly Coefficient alpha A formula that provides an estimate of the reliability of a homogeneous test or an estimate of the reliability of each dimension in a multidimensional test; A class of procedures used to estimate reliability from the internal consistency of a measure; An internal consistency reliability coefficient based on the number of parts into which the test is partitioned (e.g. items, subtests, or raters), the interrelationships of the parts, and the total test score variance; Also called Cronbach's alpha, and, for dichotomous items, KR 20 Cognitive Issues that relate to a person's ability to solve a problem using rational thought Cognitive ability The psychological concept that refers to such processes as perceiving, knowing, recognizing, conceptualizing, judging, and reasoning

Assessment Glossary


27

Cognitive ability testing Using a test instrument during the personnel selection process to measure a candidate’s learning and reasoning abilities Cognitive assessment The process of systematically gathering test scores and related data in order to make judgments about an individual's ability to perform various mental activities involved in the processing, acquisition, retention, conceptualization, and organization of sensory, perceptual, verbal, spatial, and psychomotor information Cognitive complexity The ability to differentiate, abstract, and conceptualize Cognitive development Development explained through sequential stages in which individuals encounter problems or ideas which cause cognitive conflicts that demand the individual to accommodate or change their way of thinking to become more complex Cognitive domain The range of knowledge and knowledge-related skills needed for learners to achieve different types of instructional objectives; Knowledge skills include perception, knowledge of facts, acquisition of skills, and higher-order inference Cohort A group whose progress is followed by means of measurements at different points in time; A group of study subjects, who are selected based on predetermined criteria, and who are followed over a period of time Collaboration To work together sharing ideas and resources, especially in a joint intellectual effort Collective bargaining The process by which an agent chosen by employees negotiates a formal labor agreement or settles day-to-day labor disputes on behalf of the employees in the areas of wages, benefits, working conditions, and administrative policy with parties representing the top elected or appointed executives of an organization Collective case study Studying multiple cases in one research study Collegial Participative and mutually respectful Combined model A scoring or evaluation procedure that uses features of both compensatory and conjunctive models Commercial, norm-referenced, standardized exams Exams with scores based on a comparison to a reference or normative group: Usually group administered multiple-choice "objective" tests in one or more curricular areas purchased from a private vendor Communication Human communication is a process during which source individuals initiate messages using conventionalized symbols, nonverbal signs, and contextual cues to express meanings by transmitting information in such a way that the receiving party constructs similar or parallel understandings Comparability

Assessment Glossary


28

The similarity of the phenomena (e.g. attributes, performance, assessments, data sources) being examined; The amount or degree of comparability is often used to determine the appropriateness of using one phenomenon in lieu of another and to help ensure fairness Comparative data Data from two or more similar groups which have been exposed to different conditions of the independent or intervention variable Comparative experimental model Studies that assign a program, project, or instructional material to one group of persons and compare their subsequent performance on some structured task to that of another group that was not exposed to the program, project, or instructional material Comparison group A group of subjects who are similar to the group being studied, with the exception of exposure to the independent variable; A group that provides a basis for contrast with an experimental group (the group of people participating in the program or project being evaluated); The comparison group is not subjected to the treatment or independent variable, thus creating a means for comparison with the experimental group that does receive the treatment Compatibility thesis The idea that quantitative and qualitative methods are compatible Compensation Money or another item of value given or received as payment or reparation for a service or loss Compensatory model An evaluation or scoring procedure that permits trade-offs of one attribute against another; A model in which low performance on one attribute can be offset by high performance on another; Most compensatory models have an absolute minimal level of performance for each attribute, below which trade-offs are not permitted Competence based A set of knowledge, skills, or abilities usually demonstrated on a measure Competencies What students should know, be able to do, and value; The knowledge, skills and abilities required to perform a specific task or function; See also Core competency Competency The demonstration of the ability to perform a specific task or achieve a specified criteria; A discrete unit of learning resulting from the influence of educational experiences and mastered by an individual student; A skill, knowledge, or experience that is suitable or sufficient for a specified purpose; Level at which performance is acceptable; A group of characteristics, native or acquired, which indicate an individual's ability to acquire skills in a given area; A knowledge, skill, ability, personal quality, experience, or other characteristic that is applicable to the profession of teaching Competency-based assessment Measures an individual's performance against a predetermined standard of acceptable performance; Progress is based on actual performance rather than on how well learners perform in comparison to others; See Criterion referenced assessment Competency test A test intended to establish that a student has met established minimum standards of skills and knowledge and is thus eligible for promotion, graduation, certification, or other official acknowledgment of achievement

Assessment Glossary


29

Competitive advantage Characteristics that allow an organization to outperform its rivals; What an organization does better than its competitors Competitive strategy How an enterprise competes within a specific industry or market; Also known as business strategy or enterprise strategy Complete participant Case study method where the researcher becomes a member of the group being studied without telling the members they are being studied Complete-observer Case study method where the researcher makes observations as an outsider without telling people they are being observed Completion The conclusion of an educational program and acquisition of a degree; Receipt of credential (e.g. certificate, degree) Complex-generated response An assessment that asks a student to perform or produce in order to demonstrate knowledge and skills; Such assessments will not have one right answer, but instead will result in student work across a range of quality; The assessment requires that the student engage in a task of multiple parts or steps; Scoring of the assessment involves teacher judgment based on stated criteria for performance. Component One of the parts or processes in an evaluation plan, effort, or system Composite score A single score used to express the combination, by averaging or summation, of the scores on several different tests; A score that combines two or more scores or results for the same or related attributes; A score that combines several scores according to a specified formula Comprehensive All dimensions of a learning goal with regard to scope, content, specificity, skills, and types of thinking required are addressed Comprehensive sampling Including all cases in the research study Computational skills The ability to identify problems in data, to reason numerically, and to apply and use data to solve problems Computerized adaptive test A method by which a computer selects the range of questions to be asked based on the performance of the participant on previous questions Computerized or computer assisted assessment The use of computers to measure performance on some attribute, not necessarily an attribute related to computers and technology; A common term used to describe the use of computers to support assessments

Assessment Glossary


30

Concept An abstract, general notion; A heading that characterizes a set of behaviors and beliefs Concurrent validity The relationship of one measure to another simultaneous measure or variable assessing the same or a related attribute; The means of determining a test or other assessment tool’s validity by comparing test scores against actual job performance Concurrent validity evidence Validity evidence based on the relationship between test scores and criterion scores obtained at the same time Conditioning Associating a response with a previously unrelated stimulus Confirmability The findings of the study can be confirmed by another person conducting the same study; A characteristic of objective research studies Confidence interval or limits The range around a numeric statistical value obtained from a sample within which the actual corresponding value for the population is likely to fall, at a given level of probability; The range of scores or percentages within which a population percentage is likely to be found; Usually expressed in a "plus or minus" fashion; A sample-based estimate as an interval or range of values within which the true or target population value is expected to be located (with a specified level of confidence given as a percentage); Confidence limits are the end points of a confidence interval Confidence level The specific probability of obtaining some result from a sample if it did not exist in the population as a whole; The level at or below which the relationship will be regarded as statistically significant Confidentiality Not revealing the identity of the participant to anyone other than the researcher and his or her staff; The protection of data and information from persons other than those authorized to have access; Situation in which the identity of students will not be released to other individuals or institutions beyond the teacher or others who evaluate students; Maintaining the privacy of specific data Configural scoring rule A rule for interpreting a pattern of scores on two or more assessments or parts of one assessment for the same teacher Confirmatory factor analysis Factor analysis working with a priori expectations in which specific expectations concerning the number of factors and their loadings are tested on sample data Conflict management Conflict is viewed as the interaction of interdependent people who perceive opposition of goals, aims and values, and who see the other party as potentially interfering with the realizations of these goals; Conflict resolution styles include: Competition: A win-lose strategy dealing with conflict from a dominance mind set; Accommodation: A win-lose style approaching conflict management with a more yielding mindset; Avoidance: Involves not talking about the issue in question or avoiding the other party altogether; Compromise: When both parties are willing to give and take in order to manage the conflict effectively; Collaboration: A win-win conflict management style whereby both parties work toward understanding the position of the other, in order to reach agreement

Assessment Glossary


31

Conflict of interest A situation in which an evaluator's private interests affect her or his evaluative actions, or in which the evaluative actions might affect private interests; A situation in which the private interests of someone involved in the assessment or evaluation process (e.g. interviewer, rater, scorer, evaluator) have an impact (either positive or negative) on the quality of the evaluation activities, the accuracy of the data, or the results of the evaluation Confound To confound is to fail to detect a confounding bias prior to conducting research, resulting in unreliable cause-and-effect interpretations; A confound is the confounding variable which created the bias Confounded The situation in which the effect of a controlled variable is inextricably mixed with that of another, uncontrolled variable Confounding bias A type of bias in which one or more initially unrecognized confounding variables turn out to have affected the obtained results, thus rendering cause-and-effect interpretation unreliable. Confounding variable An unforeseen and unaccounted-for variable that jeopardizes reliability and validity of an experiment's outcome; An uncontrolled variable that systematically varies with the independent variable; A type of extraneous variable that was not controlled for and is the reason a particular “confounded” result is observed; A variable which influences the dependent variable while systematically varying with the independent variable Confucian dynamism A cultural dimension that values a long-term orientation, persistence and thrift Congruence analysis The verification of data by using more than one instrument or source of data for assessing performance on the same criterion Conjunctive model An evaluation or scoring procedure that requires the student or teacher to attain a minimal level of performance on all attributes assessed. Consensus Agreement Consensus group A type of group discussion in which participants try to form a consensus on an issue Consent The granting of permission by an individual concerning the collection, use, retention, or access to assessment data and information Consequential basis of validity The assemblage of information on the theoretical and value implications of the way that the results of testing are used, and the appraisal of both the potential and actual social consequences of the testing, including side effects Consequential validity The aspect of validity dealing with positive and negative intended and unintended issues associated with the use and interpretation of tests

Assessment Glossary


32

Consistency Obtaining the same or similar results across multiple administrations or scoring of an assessment; Implementation of procedures in an identical or near identical manner across individuals or over time; A type of rater effect in which an assessor or an evaluator tends to rate or to interpret different data and information in a similar way; A consistent rater tends to assign the same grade or rating to all assessment results and products without regard to their quality or to the scoring rubric Constant A single value or category of a variable; A variable which only has a single value; A certain characteristic (like grade level) can be a variable in one study and a constant in another study Constant comparative method Data analysis in grounded theory research Constituency A group served by an organization or institution Construct Specific concept or variable of interest in a given research effort; An abstract or general idea inferred or derived from specific instances; An attribute of an individual or a phenomenon that is not directly observable, but which is theoretically based or inferred from empirical evidence; A characteristic or trait of individuals inferred from empirical evidence; The concept or characteristic that a test is designed to measure; Something that exists theoretically but is not directly observable; A concept developed for describing relations among phenomena or for other research purposes; A theoretical definition in which concepts are defined in terms of other concepts; For example: Intelligence is a construct because it is widely assumed to exist but cannot be directly observed or measured; See Hypothetical construct Construct domain The set of interrelated attributes (e.g. behaviors, attitudes, values) that are included under a construct's label; A test typically samples from a construct domain Construct equivalent The extent to which the construct measured by one test is essentially the same as the construct measured by another test; Also, the degree to which a construct measured by a test in one cultural or linguistic group is comparable to the construct measured by the same test in a different cultural or linguistic group Construct irrelevance Occurs when the assessment tool used to measure an educational or psychological construct includes items or measures extraneous, or not relevant, to the construct and that cause scores to be different from what they should be Construct response item An exercise for which examinees must create their own responses or products rather than choose a response from an enumerated set Construct under representation Occurs when some of the aspects that represent the construct to be addressed are not included in the assessment used to measure it Construct validity The degree to which a test measures what it claims or purports to be measuring; The extent to which a test or other assessment instrument measures a particular trait; The degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based; Demonstrated through patterns of inter correlations among measures with high correlations

Assessment Glossary


33

between theoretically similar measures and low correlations between theoretically dissimilar measures; Seeks an agreement between a theoretical concept and a specific measuring device, such as an observation; Used to indicate that the test scores are to be interpreted as indicating the test taker's standing on the psychological construct measured by the test; A theoretical variable inferred from multiple types of evidence, which might include the interrelations of the test scores with other variables, internal test structure, observations of response processes, as well as the content of the test; All test scores are viewed as measures of some construct, thus the phrase is redundant with validity, and the validity argument establishes the construct validity of a test; The degree to which one can infer certain constructs in a psychological theory from the test scores; The degree of fit of a measure and its interpretation with its underlying explanatory concepts, theoretical rationales, or foundations Constructavism or constructavist teaching/learning Constructavism is a learning theory that asserts that students learn best when they construct their own understanding of the world around them; Constructavist teaching is student-centered and attempts to create contexts in which students actively grapple with big issues and questions instead of being passive recipients of teacher knowledge Constructed response item or format An assessment unit with directions, a question, or a problem that elicits a written, pictorial, or graphic response from a student; An exercise for which examinees must create their own responses or products rather than choose a response from an enumerated set; Short-answer items require a few words or a number as an answer, whereas extended-response items require at least a few sentences; The examinee produces an answer or response to a given stimulus or test question; Sometimes called "open-ended"; Also known as production format Constructivist theory Posits that people build new information onto pre-existing notions and modify their understanding in light of new data, and in the process their ideas gain in complexity and power; Constructivist theorists dismiss the idea that students learn by absorbing information through lectures or repeated rote practice Consultant An individual who works independently to assist and advise client organizations with various organizational functions and responsibilities on a fee-for-service basis Consultative Conducted in a manner that solicits input from various groups, but does not require actual participation in decision-making Contamination A tendency for the assessor’s data, the scorer’s rating and judgments, or the evaluator’s conclusions to be influenced or confounded by irrelevant knowledge about the student, other personnel, or other factors that have no bearing on the student’s level of performance Content analysis Qualitative research method for analyzing written or verbal material to generate reliable evidence about a large sample; Analyzing the content of media (e.g. publications and recordings) to determine the main themes being represented, usually by counting the number of times a word or theme appears; Analysis of text documents or verbal recordings; The process of identifying and listing, in accordance with a parsimonious classification system, categories of expression contained in a variety of information sources; Usually researchers establish a set of categories and then count the number of instances that fall into each category; The crucial requirement is that the categories are sufficiently precise to enable different coders to arrive at the same results when the same body of material is examined; Requires high inter-rater reliability to be valid; May be quantitative, qualitative, or both; Typically, the major purpose of content analysis is to identify patterns Content domain

Assessment Glossary


34

The set of organized categories characterizing subject matter under which behaviors, knowledge, skills, abilities, attitudes, and other characteristics may be represented in specifications for assessment instruments by which items are classified Content learning The acquisition of knowledge from an experience, as distinct from the process of an experience Content-related evidence Validity evidence based on a judgment of the degree to which the items, tasks, or questions on a test adequately represent the construct domain of interest Content representativeness The extent to which the knowledge or skills being assessed represent an adequate sample from the domain of interest Content standards A defined domain of educational objectives; Statements that describe what students should know or be able to do within the content of a specific discipline or at the intersection of two or more disciplines; Broadly stated expectations of what students should know and be able to do in particular subjects and grade levels; Define for teachers, schools, students, and the community not only the expected student skills and knowledge, but what schools should teach; Often describe the information and skills essential to the practice or application of a particular discipline or content domain Content validity The appropriateness of the domain definition and the sampling of content, and/or the extent of congruence between the scope of a content area that an instrument or process claims to cover and what it actually does cover; Extent to which a measurement reflects the specific intended domain of content; The degree to which a test or other assessment instrument used during the selection process measures the skills, knowledge and abilities or other related job qualifications; A test has content validity if the item measures what it is intended to measure; A term used to refer to a kind or aspect of validity required when the test user wishes to estimate how an individual performs in the universe of situations the test is intended to represent Contestable markets Markets where profits are held to a competitive level due to the ease of entry into the market Context The set of circumstances or acts that surround and may affect a particular student, learning situation, classroom, or school; The environment within which the teacher works; The combination of the factors accompanying the study that may have influenced its results; For example: The geographic location of the study, its timing, the political and social climate at that time, other relevant professional activities that were in progress, and any existing pertinent economic conditions Context sensitivity Awareness by a qualitative researcher of factors such as values and beliefs that influence cultural behaviors Contextual variables Indicators or dimensions that are useful in describing the facts or circumstances that surround a particular learning situation and influence a student's performance in that situation Contextualization The identification of when and where an event took place Contingency planning

Assessment Glossary


35

The process of identifying an organization’s critical information systems and business operations in order to develop and implement plans enabling those systems and operations to resume following a disaster or other emergency situation Contingency approach The use of different administrative strategies under different conditions; The study of the relationship between factors, such as the task an agency performs or the technology it uses, and the style of supervision, type of organizational design, and other administrative strategies that will work best given those factors Contingency question An item that directs participants to different follow-up questions depending on their response Contingency table A table displaying information in cells formed by the intersection of two or more categorical variables Continuous improvement Using evaluation over time to show change for the better; cyclical evaluation of outcomes in order to repeatedly increase achievements over time Continuous quality improvement (CQI) Management technique focused on meeting or exceeding customer requirements by continuous improvement and innovation in products, processes, and services; Includes a sense of collective responsibility for learning, mutual accountability for results, habitual listening to stakeholders, data based decision making, and an ethos of continuous personal and institutional improvement Continuous variable A variable that may have fractional values (e.g. height, weight and time); Variables which can take any value and have accuracy of measurement Contrast effect A type of rater affect or artifact in which an assessor or an evaluator tends to compare one student member to another student rather than comparing that student’s level of performance to the standards Control group A group in an experiment that does not receives a treatment in order to compare the treated group against a norm; A group as closely as possible equivalent to an experimental group which is exposed to all the conditions of the investigation except for the treatment, program, project, or instructional material being studied; A group of subjects, matched to the experimental group, which does not receive the treatment of interest Convenience sampling People who are available, volunteer, or can be easily recruited are included in the sample Convergence group A group that is responsible for incorporating the important features of alternative strategies proposed by advocacy teams into a compromise strategy Convergent evidence Evidence based on the relationship between test scores and other measures of the same construct Convergent validity General agreement among ratings, gathered independently of one another, where measures should be theoretically related

Assessment Glossary


36

Conversion tables Tables used to convert a student's test scores from scale score units to grade equivalents, percentile ranks, and stanines Core competency A broad, institutional-level outcome statement describing what students are able to do at the end of their experience at the college; Core competencies are categories within which course and program level learning outcomes are classified and may not be assessed directly Core functions The major responsibilities of the unit stated in a few succinct statements; Also know as Primary Functions Core values Enduring beliefs or principles that the institution’s members hold in common and endeavor to put into action; Core values guide faculty, staff, administrators, and sometimes students in performing their work Corporate citizenship The contribution a company or organization makes to society through its core business activities, social investment and philanthropy programs, and its engagement in public policy; The manner in which a company manages its economic, social and environmental relationships and the way it engages with its stakeholders (e.g. shareholders, employees, customers, business partners, governments, and communities) and has an impact on the company’s long-term success Corporate or organizational culture The beliefs, values and practices adopted or exhibited by an organization that directly influence employee conduct and behavior Corporate governance A set of processes, customs, policies, laws, and institutions affecting the way a corporation or organization is directed, administered, or controlled Corporate image The way in which an organization is viewed by clients, employees, vendors or the general public Corporate social responsibility Concept that organizations have an obligation to consider the interests of customers, employees, shareholders, communities, and the environment in all aspects of their operations Corporate values The prescribed standards, behaviors, principles, or concepts that an organization regards as highly important Corporation A group of individuals legally empowered to transact business as one body; Legal entities created by states through the approval of the charters submitted by their founders Corrective scoring A calculation used to offset the effects of guessing in objective tests Correlation The extent or degree to which two variables are related; The degree to which two or more sets of measurements vary together; The tendency for certain values or levels of one variable to occur with particular values or levels of another variable; The degree of relationship (linear or curvilinear) between two variable, scores, or assessments; Statistical measurement of the relationship between the changes in two or more measured variables; A common statistical analysis (usually abbreviated as r) that measures the degree of

Assessment Glossary


37

relationship between pairs of interval variables in a sample and ranging from -1.00 to zero to +1.00; A non-cause and effect relationship between two variables; A positive correlation exists when high values on one scale are associated with high values on another; A negative correlation exists when high values on one scale are associated with low values on another Correlation coefficient An index that indicates the strength and direction of the relationship between two variables; A measure of association between two variables that can range from -1.00 (perfect negative relationship) to 0 (no relationship) to +1.00 (perfect positive relationship) Correlational research A form of non experimental research in which the primary independent variable of interest is a quantitative variable Corroborating evidence Comparing documents to each other to determine whether they provide the same information or reach the same conclusion; Documentation that confirms, strengthens, and/or provides support for other documentation on the same attribute, competency, or situation Cost The quantity of resources required in order to achieve a desired end Cost-benefit analysis A means of measuring the costs associated with a specific program, project, activity or benefit compared to the total benefit or value derived; The analysis of well-defined alternatives by comparing their costs and benefits when both costs and benefits are expressed in monetary terms; Each alternative is examined to see whether benefits exceed costs, and the ratios of the alternatives are compared; The alternative with the highest benefit-to-cost ratio is then selected; A management tool that involves calculating or estimating the monetary costs and potential benefits of a proposed course of action; Technique designed to measure relative gains and losses resulting from alternative policy or program options Cost-benefit ratio The proportional relationship between the expenditure of a given quantity of resources and the benefits derived their expenditure; A guideline for choosing among alternatives within the rational model of economic decision making Cost center An enterprise that has a manager who is responsible for cost performance and controls most of the factors affecting cost; A separate cost accounting unit charged with providing a method of isolating and tracking costs as a means to enhance management controls by identifying responsibility for various expenses Cost effectiveness The extent to which one program, project, or instructional material produces equal or better results than another that costs about the same amount of time, effort, and resources; The extent to which an object produces the same results as another but is less costly; Cost effective does not mean affordable because highly effective interventions may be so expensive that without additional funding they are still cost prohibitive to an institution even when they are the most cost effective Cost-per-hire Calculated measurement of the direct and indirect costs associated with filling a vacancy Counseling Intervention involving the provision of advice or support on a personal basis, by someone who has been trained to provide that support

Assessment Glossary


38

Counterbalancing Administering the experimental treatment conditions to all comparison groups, but in a different order Course An organization of subject matter and related learning experiences provided for the instruction of students on a regular or systematic basis, usually for a predetermined period of time Course or course-level assessment Applying assessment techniques to the stated learning objectives of a given course; Assessment to determine the extent to which a specific course is achieving its learning goals Course embedded assessment Collecting assessment data information within the classroom because of the opportunity it provides to use already in-place assignments and coursework for assessment purposes; Involves taking a second look at materials generated in the classroom so that, in addition to providing a basis for grading students, these materials allow faculty to evaluate their approaches to instruction and course design; Reviewing materials generated in the classroom to provide a basis for grading students and using them to evaluate approaches to instruction and course design Course objectives Statements of what the students are expected to know or learn by the end of a course; Often focus on what the instructor does rather than what the student will be able to do (i.e. focus on input rather than output); Often content-based and not necessarily competency-based; Often not measurable or assessable Course portfolio May be developed by students or faculty; Students create portfolios by gathering a body of evidence of their own learning and competences in a course; Used to help students become reflective thinkers; Faculty develop course portfolios to document their own leaning in a course and then share it with students; Course portfolios might include information illustrating how the course fits within the institution or program, as well as evidence of student learning, the instructor’s reflections and self-assessments, and perhaps the reactions of peers in the discipline Course SLOs or student learning outcomes What the student will be able to produce at the end of a course; General statement of the skills and abilities a student will possess upon successfully finishing a course; Lowest level at which SLOs are usually assessed; You do not “teach to the SLOs” but rather write the SLO to reflect what you think is most important for the student to learn or be able to accomplish; Writing course-level SLOs involves considering the overarching goals of the course, matching these goals with a particular assessment method, and articulating these overarching goals in an SLO statement; Courses may have multiple SLOs. Coursework cluster analysis Looking at the patterns of coursework that students have taken and, based on what is known about the learning outcomes associated with each of the courses a student has passed, determining whether a student is considered as having mastered the skills and knowledge necessary to graduate Covariate How one variable varies with another; Usually used in removing associated effects of an external variable such as age or family income from other variables in a study; A product of the correlation of two related variables times their standard deviations; Used in true experiments to measure the difference of treatment between the variables Covariation

Assessment Glossary


39

A state that exists when two things vary together; The degree of covariance is expressed through measures of association Creaming The process of focusing on participants who are easy to serve, with the possible consequence of neglecting those participants who are most in need of services Creativity In an educational setting, creativity pertains to the development of flexible thinking patterns; Entails aspects of problem solving and motivation to find solutions as well as the generation of multiple hypotheses Credibility A researcher's ability to demonstrate that the object of a study is accurately identified and described, based on the way in which the study was conducted; Believability or confidence by virtue of being trustworthy and possessing pertinent knowledge, skills, and experience Credible Worthy of confidence and acceptance by others; Usually based on the expertise, trustworthiness, and/or reliability of the source of the evidence or judgment; Credible does not necessarily mean accurate or valid Crises management The practice of implementing organization-wide policies and procedures containing pre-established guidelines for responding to catastrophic events or tragedies (e.g. fires, earthquakes, severe storms, workplace violence, kidnappings, bomb threats, and acts of terrorism) in a safe and effective manner Crises planning Creating formal written plans that establish specific measures or actions to be taken when responding to catastrophic events or tragedies in the workplace Crises prevention The process of an organization implementing specific plans and procedures designed to circumvent certain disasters or emergencies Criteria or standards Characteristics of good performance on a particular task; Guidelines, rules, characteristics, dimensions, or principles by which student responses, products, or performances are evaluated; Description of performance or other indicators that demonstrate how students or service units will meet the expectations stated in the outcomes; Descriptive benchmarks against which performance is judged; A dimension along which performance (e.g. effective teaching) is rated or judged as successful or meritorious; Qualitative or quantitative statements used to measure whether the program standard of competency achievement has been met; Criteria indicate what we value in student responses, products or performances; May be holistic, analytic, general, or specific; May be described in terms of measures such as rubrics or grades; Often stated in terms of percentages, percentiles, or other quantitative measures; Sometimes called performance standards Criterion The singular of criteria; A behavior, characteristic, or quality of a product or performance about which some judgment is made; What a teacher intended to teach and what is checked to see if students did indeed learn what the teacher thought was being taught; A dimension along which performance is rated or judged as successful or meritorious; A standard by which something can be judged; The standard or benchmark that you want to predict accurately on the basis of the test scores Criterion domain The construct domain of a variable used as a criterion

Assessment Glossary


40

Criterion of falsifiability The property that statements and theories should be refutable Criterion referenced assessment Performance interpreted in relation to pre specified standards; Assessment where an individual's performance is compared to a specific learning objective or performance standard and not to the performance of other students; Assessments which determine what the assessed can do and what they know, not how they compare to others; Compares a student's performance to specific standards; Reports on how well students are doing relative to a predetermined performance level on a specified set of educational goals or outcomes included in the curriculum; Tells us how well students are performing on specific goals or standards rather than just telling how their performance compares to a norm group of students nationally or locally Criterion-referenced measurement Measurement that provides data on the extent to which particular objectives or criteria are achieved Criterion referenced score interpretation A score interpretation that does not depend upon the score's rank within, or relationship to, the distribution of scores for other examinees Criterion referenced tests Tests whose scores are interpreted by referral to well defined domains of content or behaviors, rather than by referral to the performance of some comparable group of people; A test in which the results can be used to determine a student's progress toward mastery of a content area; Performance is compared to an expected level of mastery in a content area rather than to other students' scores; A measurement of achievement of specific criteria stated as levels of mastery; Test that allows its users to make score interpretations in relation to a functional performance level, as distinguished from those interpretations that are made in relation to the performance of others; Examples include comparisons to cut scores, interpretations based on expectancy tables, and domain-referenced score interpretations; A measurement of achievement of specific criteria or skills in terms of absolute levels of mastery; The focus is on performance of an individual as measured against a standard or criteria rather than against performance of others who take the same test, as with norm-referenced tests Criterion related validity The correlation or extent of agreement of the test score from an assessment with one or more external variables that measure the attribute being assessed; Used to demonstrate the accuracy of a measuring procedure by comparing it with another procedure which has been demonstrated to be valid; Also referred to as instrumental validity Criterion-related validity evidence Validity evidence based on the extent to which scores from a test can be used to predict or infer performance on some criterion such as a test or future performance Critical case sampling Selecting what are believed to be particularly important cases Critical ethnography A style of discourse and analysis embedded within conventional ethnography in which the researcher chooses between conceptual alternatives and value-laden judgments to challenge research, policy, and other forms of human activity; Critical ethnographers attempt to aid emancipation goals, negate repressive influences, raise consciousness, and invoke a call to action that potentially will lead to social change Critical incident A significant and observable episode or performance (effective or ineffective) in a student’s career that alters the direction of subsequent behaviors, activities, or events

Assessment Glossary


41

Critical incident appraisal The use of documentation concerning critical incidents when evaluating and making decisions about a student’s current and potential performance Critical information Knowledge about the subject being evaluated or the assessment instruments, evaluation process, and/or working conditions and learning/teaching context that must be known to avoid incorrect interpretations of results Critical reflection Action research process where participants and researchers in a project think back over a project, critically comment on what has happened in the project, and then use the new learning to improve future activities Critical research paradigm Focuses on a critical understanding of the situation or practice being researched in order to plan for transformative action; The critical research approach is situated in the context of actual people and practices and emphasizes social change Critical success factors The key items that must be met in order to successfully achieve a specific objective; Those few things that must go well if a project or organization is to succeed; Also called key success factors Critical thinking The ability to integrate previous knowledge to identify and weigh possible responses to a novel situation; Ability defined by the seven major categories of interpretation, analysis, evaluation, inference, presenting arguments, reflection, and dispositions; Within each of these categories are sets of skills and sub skills Cronbach's alpha One the measures of internal consistency that falls under the broad heading of coefficient alpha Cross-case analysis Searching for similarities and differences across multiple cases Cross-sectional research A study, which measures a population at a specific point in time or over a short period of time; An alternative to longitudinal research Cross-sectional survey Involves observations of a sample, or cross-section, of a population or phenomenon that are made at one point in time Cross-tabulation, crosstabs, cross partitions, or crossbreaks A way of arranging data about categorical variables in a matrix so that relations can be more clearly seen Cross validation The application of a scoring system or set of weights empirically derived in one sample to a different sample drawn from the same population to investigate the stability of relationships based on the original weights Crosswalk A research tool used to guide analysis and reporting, particularly when there are multiple data sources; A crosswalk does not present any of the findings or results, just the types of information that has been gathered from the different data sources

Assessment Glossary


42

Cultural competence A set of values, behaviors, attitudes, and practices which enable people to work effectively across racial/ethnic and cultural lines Cultural differences The diverse set of behaviors, beliefs, customs, traditions, language, and expressions characteristic of people who are members of the same group; Often associated with groups of people defined by a particular race, ethnicity or national origin Cultural relativity or relativism The perspective that the appropriateness of any behavior must be evaluated from the cultural context of its occurrence; Opposite of ethnocentrism, which interprets and judges the behavior of others from the perspective of the observer’s culture; Technique used by anthropologists to control for their own ethnocentrism (or observer bias) when studying other cultures and interpreting the behavior of individuals from other cultures; Principle that an individual human's beliefs and activities should be understood in terms of his or her own culture; Studying another culture from its point of view without imposing our own cultural views; Most commonly misinterpreted and falsely defined as the belief that there are no universal principles of right or wrong because the meanings of ‘right’ and ‘wrong’ are solely determined by one’s culture; Often misused to dismiss without valid justification those values which are inconvenient to achieving the goals of an individual or group Cultural sensitivity An acceptance and appreciation for the customary beliefs, social forms, and material traits of different racial, religious, or social groups as well as of one’s own cultural group Culture A system of shared beliefs, values, practices, perspectives, folk knowledge, language, norms, rituals, and material objects and artifacts that the members of a group use in understanding their world and in relating to others; A group’s shared assumptive reality; The shared values, understandings, assumptions and beliefs held by members of a certain group or society that produce the norms that shape the behavior of people in that group; The collection of beliefs, expectations, and values learned and shared by group members and passed on from one generation to another Culture-fair test A test devised to exclude specific cultural stimuli so that persons from a particular culture will not be penalized or rewarded on the basis of differential familiarity with the stimuli Culture of assessment An environment in which continuous improvement through assessment is expected and valued Culture of evidence An environment in which using research and/or assessment results to guide policy and management decisions is expected and valued Curricular validity The extent to which the items on the assessment or test measure the content of a local curriculum, or the extent of agreement between the test coverage (topics, breadth and depth, skills, cognitive complexity) and the goals and objectives of the curriculum Curriculum The knowledge, skills, attitudes, behaviors, and values students are expected to learn from schooling; Includes statement of expected student outcomes, descriptions of material and activities, and the planned sequence that will be used to help students acquire the expected outcomes; A comprehensive overview, including activities planned for delivery to the students, the scope of content, the sequence of materials, interpretation and balance of subject matter, and motivational, instructional, and assessment techniques to be used; A set of ordered

Assessment Glossary


43

intended learning outcomes; All of the instruction, services, and activities provided for students through formal schooling including but not limited to: content, teaching methods and practices, instructional materials and guides, the physical learning environment, assessment and evaluation, time organization, leadership, and controls; Curriculum includes planned, overt topics of instruction as well as unseen elements such as norms and values taught by the school and through classroom interactions between the teacher and learner, hidden social messages imbedded in the curriculum materials themselves, and the material that is not included in the overt or planned curriculum Curriculum alignment The degree to which a curriculum's scope and sequence matches a target outcome or assessment measure Curriculum embedded or learning embedded assessment Assessment that occurs simultaneously with learning, such as by completing projects, portfolios and exhibitions; Occurs in the classroom setting with students unable to tell whether they are being taught or assessed; Assessment tasks or tests developed from or inherent within the curriculum or instructional materials Curriculum framework A document outlining content strands and learning standards for a given subject area which provides a structure from which lessons and curricula can be organized and presented to the student Curriculum map or mapping A matrix relating program-level student learning objectives or outcomes (usually enumerated in individual rows) to the courses and/or experiences that students take in progress to graduation (usually captured in columns); A matrix showing the coverage of each program learning outcome in each course; May also indicate the level of emphasis of each outcome in each course; A matrix representation of a program's learning outcomes showing where the outcomes are taught within the program Cut score or critical score A score which a student needs to achieve to demonstrate minimal competency; A specified point in a predictor distribution of scores below which candidates are rejected or considered not to have reached a minimum standard of performance; Score used to determine the minimum performance level needed to pass a competency test; A specified point on a score scale, such that scores at or above that point are interpreted or acted upon differently from scores below that point Cutting score A score that marks the difference between two levels of teaching performance (e.g. good and excellent); When the difference is between minimally acceptable and not acceptable, or pass and not pass, it is referred to as a cut score, critical score, or passing score Cycle Collecting data and learning from it over and over; A round of assessment, evaluation/analysis, intervention/planning and communication which is repeated

Assessment Glossary


44

D Data Discrete units of information, usually numbers; Recorded observations, usually in numeric or textual form; Information collected by a researcher; The information and evidence gathered during the assessment process for use in determining the level of learning outcomes Data access Conditions under which access to information is provided, including who has the access; Extent to which the evaluator will be permitted to obtain data during the course of an evaluation Data analysis Process of organizing, summarizing, and interpreting numerical, narrative, or artifact data, so that the results can be validly interpreted and used to guide future development of students Data acquisition skills The ability to use references (including the Internet) to find information and to judiciously select valid data sources Data collection procedures Set of steps used to obtain quantitative or qualitative information about the knowledge, skills, attitudes, or behaviors possessed by a student Data integration The merging of related data for use in scoring, judging, and evaluating Data mining The use of technological applications to aid and support decision making by locating, manipulating and synchronizing information buried deep within corporate databases Data set A set of data; A collection of related data items Data sources Persons, documents, products, activities, events, and records from which the data are obtained Data triangulation The use of multiple data sources Data warehouse A system used to collect operational data from different functions, areas or business units into a central database Database A collection of information/data, often organized through tables, within a computer's mass storage system; Databases are structured in a way to provide for rapid search and retrieval by computer software De-layering An organizational restructuring strategy meant to reduce the organization’s existing levels of managers or supervisors

Assessment Glossary


45

Debriefing A post study interview in which all aspects of the study are revealed, any reasons for deception are explained, and any questions the participant has about the study are answered; The simplest form is a short recapitulation of what subjects have done and why they have done it; Part of the research procedure which is an important aspect of ethicality in research; See also dehoaxing Deception Misleading or withholding information from the research participant; The deliberate concealment of the true purpose of a piece of research, often aided by a cover story delivered at the briefing; Generally only considered ethical if the deception is necessary to avoid demand characteristics, reduce experimenter effects, or otherwise control confounding Decentralization The process of assigning decision-making authority to lower levels within the organizational hierarchy Decile Any one of the nine points (scores) that divide a distribution into ten parts, each containing one-tenth of all the scores of cases; Every tenth percentile (the first decile is the 10th percentile, the second decile is the 20th percentile, the third decile the 30th percentile, etc.) Decision consistency coefficient Similar to the reliability coefficient; A calculated value that tells the extent to which the decision results (e.g. classifications) would be the same if the process were repeated; Often used in either-or decision situations (e.g. mastery-non mastery); A coefficient of zero (0) means no consistency and a coefficient of one (1.0) means fully consistent Decision tree Technique that identifies various possible outcomes given the risks associated with each decision option Decision rules Rules for choosing between optional interpretations or courses of action given certain evidence; For example: A rule by which teachers pass or fail students in a course based on their test scores and other performances in the course, or a rule by which an evaluator decides that the difference between the test scores of students exposed to different programs is statistically significant Deductive reasoning The process of drawing a specific conclusion from a set of premises; A form of reasoning in which conclusions are formulated about particulars from general or universal premises; The ability to extract certain rules based on a sequence of experiences or observations and apply those rules to other similar situations; Deriving conclusions and hypotheses from the data; Data is collected and analyzed before assuming a hypothesis Deductive method A top-down or confirmatory approach to research Defensible An action, conclusion, or statement that is explainable or justifiable, based on a solid foundation and policy, explicit evaluation procedures, valid assessments, and sound evaluation and decision-making practices Degree A title conferred on students by a college, university, or professional school on completion of a program of study Degrees of freedom (df)

Assessment Glossary


46

The number of components of a statistic which are free to vary; A mathematically complex concept which is usually simple to determine and use (e.g. the degrees of freedom for a variable sampled at n different intensities is simply n-1) Dehoaxing Informing study participants about any deception used and the reasons for its use; Important component of debriefing when the research involved any intentional deception Delegation The process of assigning tasks or projects to subordinates and clearly dictating expected outcomes and time frame for completion Deliberative poll A method of obtaining informed polling data where a group of randomly selected citizens meet, discuss an issue for some time, and then vote on the issue Demand The number of units of a particular product or service that can be sold at the price the company is planning to charge Demand characteristics Confounding variable resulting from the experimental set-up itself, including the behavior or appearance of the researcher(s); Subtle environmental factors which interact with the motivational state of subjects during the research experience to make the observed behavior non-natural in some important respect; The demonstrable fact that the research setting may evoke behavior you did not intend to evoke and which is not a result of the independent variable Demand forecasting A strategy intended to project an organization’s anticipated workloads over a specific period of time in order to determine future personnel needs Demographics The characteristics of human populations and population segments, especially when used to identify consumer markets Demonstrated competence A student who has shown the ability to perform a specific task or skill at a specified level using some predetermined measure Deontological approach An ethical approach that says ethical issues must be judged on the basis of some universal code Departmentalization The process of dividing an organization’s labor, functions, processes or units into separate groups Dependability Being able to account for changes in the design of the study and the changing conditions surrounding what was studied; A measure of how consistent the results obtained in an assessment are in a criterion-referenced evaluation; Consistency of decisions in relation to pre specified standards Dependent variable (DV) A variable that is presumed to be influenced by one or more independent variables; A variable that is considered to be an effect or a variable that is predicted; A variable that receives a stimulus or treatment and is measured for the effect the treatment has had on it; A measure that is assumed to vary as a result of some influence (often

Assessment Glossary


47

referred to as the independent variable); In experimental research, the researcher assumes the dependent variable or factor is caused by (or depends upon) another variable (the independent variable); Sometimes called the outcome variable Delphi technique A method for obtaining group consensus involving the use of a series of mailed questionnaires and controlled feedback to respondents which continues until consensus is reached Deployment How a system utilizes tools throughout the operation Depth or in-depth interview Qualitative research technique which uses long, probing interviews without a formal questionnaire; Also known as a depth Depth of knowledge The extensiveness of knowledge and understanding within a certain area of content Derived score A score to which raw scores are converted by numerical transformation (e.g. conversion of raw scores to percentile ranks or standard scores); A test score pertaining to a norm group (such as a percentile, stanine, or grade equivalent) that is an outgrowth of the scale scores; May be useful as descriptors, but because they are not calibrated on an equal-interval scale they cannot be added, subtracted, or averaged Description Attempting to describe the characteristics of a phenomenon Descriptive research Research focused on providing an accurate description or picture of the status or characteristics of a situation or phenomenon Descriptive statistics Statistics that focus on describing, summarizing, or explaining data; A way of summarizing data by letting one number stand for a group of numbers; Set of mathematical procedures designed to present research data in summary form which is not part of hypothesis testing (e.g. mean, median, mode, range, and standard deviation); Commonly presented in tables and graphs to summarize data; Most common graphical displays are bar charts, box-and-whisker charts, histograms, and pie charts Descriptive validity The factual accuracy of an account as reported by the researcher Descriptors Statements of expected performance at each level of performance for a particular criterion in a rubric; A set of signs used as a scale against which a performance or product is placed in an evaluation; Allows assessments to include clear guidelines for what is and is not valued in student work; Typically found in analytic rubrics Desensitizing Helping study participants deal with and eliminate any stress or other undesirable feelings that the study might have created Design A plan related to operations; Organization and structuring of resources and actions to achieve a desired end outcome; A representation of the set of decisions that determine how a student evaluation is to be conducted (identifying purposes and use of the information, developing or selecting of assessment methods, collecting

Assessment Glossary


48

assessment information, judging and scoring student performance, summarizing and interpreting results, reporting evaluation findings, and following up on evaluation results); A plan for conducting an evaluation (data collection schedule, report schedules, questions to be addressed, analysis plan, management plan, etc.); Designs may be either preordinate or emergent Design flexibility A quality of an observational study that allows researchers to pursue inquiries on new topics or questions which emerge from the initial research Desired outcomes The results or products that a training program, process, instructional unit, or learning activity strives to achieve, as defined in measurable terms; See also Goals and Objectives Determinism The proposition that all events have causes Development The progression of an individual through various phases of life Developmental assessment Evaluates students throughout their undergraduate experience rather than only at the end of their program; A true longitudinal approach would evaluate students at several points in the program, providing baseline information about students’ knowledge and skills, and identifying changes in students’ growth and development Developmental delay Failure of a developing organism to reach, achieve, or display some physical, cognitive, or behavioral developmental norm at the expected chronological age Developmental norm Age-related expectation of mental or physical ability derived from prior experience or research with the parent population; Vitally important in detecting developmental delay and part of the standardization process required for major psychometric tests; See Norm Developmentally appropriate Characteristic of an assessment task that reflects the skills and knowledge which teachers and students, with a given level of training and experience, have had a reasonable chance of acquiring or learning Deviation Short for deviation from the mean; The distance between the mean and a particular data point in a given distribution Diagnostic test Tests designed to identify where a student is having difficulty with an academic skill; A test used to analyze or locate an individual's specific areas of weakness or strength, to determine the nature of his weaknesses or deficiencies, and, wherever possible, to suggest their cause; A test intended to locate learning difficulties or patterns of error; May be used to identify a need for remedial instruction; Primarily used to identify needs and to determine prior knowledge of participants; Usually occur prior to a learning experience; An intensive, in-depth evaluation process with a relatively detailed and narrow coverage of a specific area; Usually the purpose of this test is to determine the specific learning needs of individual students and to be able to meet those needs through regular or remedial classroom instruction; Measurements and measurement packages designed to assist the assessment phase of patient or student management; Required to have demonstrated or known sensitivity (ability to detect someone who needs to be detected), specificity (ability to exclude people who need to be excluded), positive predictive value (a measure of how many of those detected as positive actually are

Assessment Glossary


49

positive), and negative predictive value (a measure of how many of those detected as negative actually are negative); Examples include intake assessments, placement tests, and entrance exams Diagramming Making a sketch, drawing, or outline to show how something works or to clarify the relationship between the parts of a whole Dialogue Self-reflective exchanges engaged in by the college community and characterized by a free exchange of ideas without the purpose of defending or deciding on a course of action; Group discussions, often with facilitation, among colleagues designed to explore complex issues, increase group intelligence, and facilitate group learning; An essential process in SLO development and assessment; ACCJC accreditation theme Differential attrition A differential loss of participants from the various comparison groups; Participants who drop out are different from those who stay in the study Differential functioning Characteristic of an assessment approach, task, instrument, or evaluation system that yields higher results for one group than another group, even though both groups have the same level of ability or competence on that aspect of teaching Differential impact The varying impact of policy or research on different groups; Impact which is not systematic across groups Differential influence When the influence of an extraneous variable is different for the various comparison groups Differential item functioning (DIF) A statistical procedure used to identify items that function differently for different groups; Not just item bias Differential prediction Extent to which a measure estimates future performance on the same attribute differently for two or more groups of teachers who vary on relevant characteristics, such as years of teaching experience or special training completed Differential selection Selecting participants for the various treatment groups that have different characteristics Differentiated instruction A model of instruction that responds to the different readiness levels of students in the classroom; It supports a more individualized education for students by creating a variety of activities and assignments that meet students’ diverse needs Difficulty or facility A statistical property indicating the level of a question from 0.0 to 1.0; Calculated as the average score for the question divided by the maximum achievable score; A facility of 0.0 means that the question is very hard (no-one got it right), 1.0 means that it is very easy (no-one got it wrong), and 0.5 is considered ideal Diffusion curve The rate over time at which innovations are copied by competitors Digital divide

Assessment Glossary


50

The increasing differentiation between groups or categories of people resulting from the differing ability of their members to utilize recent technological advances to their own benefit Dimensions Aspects or categories in which performance in a domain or subject area will be judged; Separate descriptors or scoring methods may apply to each dimension of the student's performance assessment; Desired knowledge or skills measured in an assessment which are usually represented in a scoring rubric; For example, a measurement of student teamwork skills on a performance assessment might include 6 dimensions: adaptability (recognizing problems and responding appropriately), coordination (organizing team activities to complete a task on time), decision making (using available information to make decisions), interpersonal (interacting cooperatively with other team members), leadership (providing direction for the team), and communication (clearly and accurately exchanging information between team members) Direct assessment Method of gauging student achievement of learning outcomes through evaluation of student work products; Gathers evidence about student learning based on student performance that demonstrates the learning itself; Can be value added, related to standards, qualitative or quantitative, embedded or not, and/or using local or external criteria; Examples include written assignments, classroom assignments, presentations, test results, projects, logs, portfolios, and direct observations Direct costs The costs directly attributed to a particular product, program, service, or activity Direct evidence, measures, or methods Evidence that shows directly that a student has learned; Evidence produced directly by students; Measures of student leaning requiring students to display or demonstrate their knowledge and skills as they respond to the instrument; Processes employed to assess student learning directly by requiring students to demonstrate knowledge and skills; Students or learners display knowledge and skills as they respond directly to the instrument itself; Examples include essays, tests, portfolios, and demonstrations Direct measures of learning See direct evidence Directional alternative hypothesis An alternative hypothesis that contains either a greater than sign (>), or a less than sign (<) Disaggregation The division of a collective body into its component parts Disaster recovery plan A set of guidelines and procedures to be used by an organization for the recovery of data lost due to severe forces of nature such as earthquakes, fires, tornadoes, floods or hurricanes Discourse community A community of scholars and researchers in a given field who respond to and communicate to each other through published articles in the community's journals and presentations at conventions; All members of the discourse community adhere to certain conventions for the presentation of their theories and research Discrepancy Difference in results between two or more raters or scorers on the same assessment, or between two or more evaluators concerning the same teacher Discrete variable

Assessment Glossary


51

A variable that is measured solely in whole units (e.g. gender and siblings); Variables which can have only certain values (e.g. whole numbers); One of the two sub-classes of interval/ratio data; See also Continuous variable Discriminant analysis A statistical technique which allows the researcher to categorize differences between two or more groups with respect to several variables simultaneously; One of the four types of multivariate methods Discriminant evidence Evidence that the scores on your focal test are not highly related to the scores from other tests that are designed to measure theoretically different constructs Discriminate validity The lack of a relationship among measures which theoretically should not be related Discrimination index An index that indicates how well an item distinguishes between the students who understand the content being assessed and those who do not; Positive discrimination indicates that the item or task is discriminating in the same way as the assessment method of which it is a part Discrimination parameter The property that indicates how accurately an item distinguishes between examinees of high ability and those of low ability on the trait being measured; An item that can be answered equally well by examinees of low and high ability does not discriminate well and does not give any information about relative levels of performance Disjunctive model An evaluation or scoring procedure that requires the student or teacher to achieve a minimal level of performance on only one of the attributes assessed Disordinal interaction effect An interaction effect that occurs when the lines on a graph plotting the effect cross Dispersion A measure of how tightly a distribution is clustered around its mean Disproportional stratified sampling Type of stratified sampling in which the sample proportions are made to be different from the population proportions on the stratification variable Dissemination The communication (written, oral, and/or other) of the actions of evaluators, assessment outcomes, or audit results to all right-to-know audiences in order to foster knowledge of the findings Distance or distributed learning Education which occurs outside of the traditional classroom setting Distinctive competence Special skills and resources that generate strengths that competitors cannot easily match or imitate Distractors The incorrect alternatives or choices in a selected response item Distribution or frequency distribution

Assessment Glossary


52

The range of values of a particular variable; A tabulation of the scores (or other attributes) of a group of individuals to show the number (frequency) of each score, or of those within the range of each interval Diversification A strategy to increase the types of business, services or products a company is providing Documentation Proof of an event; Evidence that something happened; Collection or compilation of all tangible materials, records, and forms used in an assessment or evaluation; Naturalistic assessment process which involves recording classroom observations over time, across learning modalities, and in coordination with colleagues; The body of literature (e.g. tests manuals, manual supplements, research reports, publications, user's guides, etc.) made available by publishers and test authors to support test use Documenting The process of recording and providing tangible evidence and information about the performance of an individual, course, program, department, treatment, institution, process, or other entity Domain-referenced test A test that allows users to estimate the amount of a specified content domain that an individual has learned; May be based on sets of instructional objectives; A test in which performance is measured against a well-defined set of tasks or body of knowledge (domain); Domain-referenced tests are a specific set of criterion-referenced tests and have a similar purpose Dotted line relationship An organizational structure whereby a person reports to one manager but also has responsibilities to another Double blind A research design in which both the experimenters and participants are ignorant of the true purpose of the research; Used when experimenter effects or other factors might bias the results a study Downsizing The process of reducing the employer’s workforce through the elimination of positions, management layers, processes, functions, etc. Dropout A person who does not complete a study or program Dynamic information Information that is characterized by continuous change, information activity, or progress Dynamic systems Systems in motion; Most dynamic systems, like all living systems, are open systems; A complex interactive system evolving over time through multiple modes of behavior and following certain rules (e.g. the cardio vascular system); Qualitative observational research of dynamic systems is not concerned with having straight-forward, right or wrong answers, thus change in a study is common because the researcher is not concerned with finding only one answer Dynamic or dynamical systems theory An area of mathematics used to describe the behavior of complex systems by employing differential and difference equations

Assessment Glossary


53

E Ecological validity The ability to generalize the study results across settings Economic benefit The amount of money an individual receives from working; The gains in income or profits over time due to an action or decision Economic indicator Statistical data representing a broad range of economic trends Economic return Financial benefit of higher paying employment as a result of educational attainment; Financial benefit of an investment of resources (e.g. time, effort, or money) Economies of integration Cost savings generated from joint production, purchasing, marketing, or control Economies of scale or size A theory that as a company gets larger, its operations and associated unit costs decrease; Production costs per unit decreases as the number of units produced increases; When increased levels of production result in decreased average costs of production Economies of scope Cost reductions allowed by the products of two or more enterprises being produced from shared resources Education-to-education Programs and services provided by a learning-dedicated environment to facilitate learning and development through a successful transition of students from school to further education Education-to-work Through contextual, applied, and focused educational experiences students are provided with the knowledge and skills for employment Educational objective A statement describing the knowledge, skill, attitude, or behavior a student is expected to learn or perform and the content on which it will be performed as a result of instruction Educational providers Institution, whether university, corporate, or virtual, whose programs or services produce student learning or development Educational success Increasing student learning and/or development; Meeting competency; Obtaining credit, credential, or degree Effect size The difference between the mean of the experimental groups and the mean of the control groups in standard deviation units Effect size indicator

Assessment Glossary


54

A measure of the strength of a relationship Effective An attribute of institutions, teachers, programs, and approaches that meet their stakeholders' needs; The mark of effectiveness in higher education is desirable change in students Effectiveness Extent to which an individual, program, activity, or organization is achieving or failing to achieve its stated objectives; A measure of the likely actual benefit arising from a given remediation program or intervention Effects Measures of the change in response produced by a change in the level of a factor Efficacy A measure of the theoretically maximum benefit arising from a given remediation program or intervention under ideal conditions Efficiency Relationship between inputs and outputs; Resource management indices Efficient Producing more educational output per dollar; An attribute of institutions, teachers, programs, and approaches that balance effectiveness against considerations of costs (i.e., are effective with a minimum use of resources) Elasticity of demand The change, expressed as a percentage, in the quantity of demand for a product or service divided by the change in the price being charged Electronic portfolio or e-portfolio A collection of work developed across varied contexts over time saved in an electronic format; May advance learning by providing students and/or faculty with an easier way to organize, archive and display pieces of work; Increasingly popular because they offer practitioners and peers the opportunity to review, communicate and assess portfolios in an asynchronous manner Element The basic unit that is selected from the population Email or e-mail discussion group A discussion group whose messages are distributed by automated, mass-distributed email programs Embedded assessment Means of gathering information about student learning that is built into and a natural part of the teaching-learning process; Often uses for assessment purposes classroom assignments that are otherwise evaluated to assign students a grade; Can assess individual student performance or aggregate the information to provide information about the course or program; Can be formative or summative, quantitative or qualitative (e.g. an existing course research paper is graded for content and style as part of the course grade, but it is also assessed for ability to locate and evaluate Web-based information as part of a college-wide information literacy SLO assessment); Collecting assessment data information within the classroom using in-place assignments and coursework; Using materials generated in the classroom to allow faculty to evaluate instruction techniques and course design in addition to providing the basis for grading students; Embedded assessments need not be a part of the student’s grade; Collecting assessment data information within the classroom because of the opportunity it provides to use already in-place assignments and coursework for assessment purposes; For example: Including questions for assessment purposes within existing course exams; Involves taking a second

Assessment Glossary


55

look at materials generated in the classroom so that, in addition to providing a basis for grading students, these materials also allow faculty to evaluate their approaches to instruction and course design Emergency planning The process of establishing specific measures or actions to be taken when responding to catastrophic events or tragedies (i.e. fire, earthquake, severe storms, workplace violence, kidnapping, bomb threats, acts of terrorism, or other emergency situations) in the workplace Emergent design An implementation plan in which the specification of every step depends upon the results of previous steps; Sometimes also known as a cascading or rolling design Emic The intrinsic differences within a culture which are meaningful to group members Emic perspective The insider’s perspective Emic term A special word or term used by the people in a group Empathic neutrality A quality of qualitative researchers who strive to be non-judgmental when compiling findings Empirical Based on observation, experiment, or experience Empirical data Data obtained by actual observation rather than by conjecture; Data from the evidence of the senses Empirical research The process of developing systematized knowledge gained from observations that are formulated to support insights and generalizations about the phenomena under study Empirical statement A statement based on observation, experiment, or experience Empiricism The idea that knowledge comes from experience Employer of choice A term used to describe a public or private employer whose practices, policies, benefits and overall work conditions have enabled it to successfully attract and retain talent because employees choose to work there Employer satisfaction Employer approval of the worksite performance of educational/technical program completers in their employ Empowerment evaluation An evaluation approach that includes collaborative and training functions within a goal of the empowerment of management and program staff to continuously assure quality of services Enabling techniques A set of methods usually used in focus groups to help people produce ideas and give their opinions indirectly

Assessment Glossary


56

Enlightenment effects Type of confounding where the possibility that prior exposure to the study area might influence the behavior being tested Enterprise resource planning (ERP) A series or set of computer applications used to help monitor and manage the operations of an enterprise Entitlement grants Grants that provide assistance to persons who meet certain criteria Entitlement programs Programs that provide a specified set of benefits to those who meet certain eligibility requirements Entry level Information collected by programs about potential students to determine their qualifications for entering a program; In some cases, information is also used to select the appropriate level for students to begin their studies; Information collected for entry-level decisions about individual students can provide a good starting point for assessing academic programs; Lowest level of employment in an industry or career path Enumeration The process of quantifying data Environmental analysis An analysis of the environmental factors that influence a firm’s operations Environmental opportunity An attractive area for a firm to participate in where the firm would enjoy a competitive advantage Environmental scanning A review process of examining environmental factors at a given point in time and in a specific setting; A process that systematically surveys and interprets relevant data to identify external opportunities and threats; To monitor, evaluate, and disseminate information from the external environment to key people within an organization Environmental threat An unfavorable trend or development in the firm’s environment that may lead to an erosion of the firm’s competitive position Envisioning or visioning A set of qualitative techniques used for helping organizations or communities clarify their visions of desirable futures Epistemology The study of how knowledge is generated and justified; How something is learned and believed as true Equal employment opportunity Refers to efforts to eliminate employment discrimination on the basis of race, ethnic background, sex, age, or physical handicap; Ensures that all persons have an equal chance to compete for employment and promotions based on job qualifications Equal-interval scale A scale marked off in units of equal size that is applied to all groups taking a given test, regardless of group characteristics or time of year

Assessment Glossary


57

Equal probability selection method Any sampling method where each member of the population has an equal chance of being selected Equating A statistical process used to convert scores on two or more alternate forms of an assessment instrument to a common score for purposes of comparability and equivalence; Statistically adjusting the scores of a test to make scores from two tests comparable Equitable That which is fair, impartial, and just, and which provides equal opportunity for all Equity A quality or state that is fair, impartial, and just; A criterion for allocating resources on the basis of fairness; Concern for fairness; Assessments are free from bias or favoritism; Assessments are reviewed for stereotypes, situations that may favor one culture over another, excessive language demands that prevent some students from showing their knowledge, and the potential to include students with disabilities or limited English proficiency Equity of opportunity Providing all students with the same opportunity to succeed Equity of outcomes Ensuring that all students meet or exceed the same high standard Equivalence The comparability of two or more parallel measures that have been designed to assess the same aspect of learning/teaching and to yield similar evaluation results regardless of the measure used or the scoring/rating procedure applied; Two non-identical items which have the same meaning Equivalency reliability The extent to which two items measure identical concepts at an identical level of difficulty Equivalent-forms reliability The consistency of a group of individuals’ scores on two equivalent forms of a test measuring the same thing Error The difference between a person’s true score and his or her observed score; The extent to which a score, assessment, or calculation is incorrect or inaccurate; Difference between an observed score and a predicted or estimated score Error of measurement The difference between an observed score and the corresponding true score or proficiency; The difference between a learner/teacher's obtained score and his/her true score on an assessment that is due to factors beyond the control of that teacher; Such factors include lack of reliability in the assessment instrument or process, variability of settings of the assessment, limited sampling of teacher performance, bias of the assessor, rater effects, and interactions among such factors Error variance The estimate of variance of the distribution of the population of individual cases based on the variation among the scores within a group sample; Same as within group variance Essay test

Assessment Glossary


58

A test that requires students to answer questions in writing; Responses can be brief or extensive; Tests for recall and ability to apply knowledge of a subject to questions about the subject, rather than ability to choose the least incorrect answer from a menu of options Estimate An indication of the value of an unknown quantity based on observed data; An approximation of a true score, parameter, or value; The particular value of an estimator that is obtained from a particular sample of data and used to indicate the value of a parameter; Because no instrument or statistical procedure can provide an exact (or true) score or value, essentially all data are estimates; The smaller the error of measurement, the more precise the estimate of the true score or value Estimation Estimation is the process by which sample data are used to indicate the value of an unknown quantity in a population; Results of estimation can be expressed as a single value (point estimate) or a range of values (known as a confidence interval) Estimator

An estimator is any quantity calculated from the sample data which is used to give information about an unknown quantity in the population (e.g. the sample mean is an estimator of the population mean) Ethical Performing the evaluation or behaving in accordance with a moral code of conduct that addresses such issues as the well-being of the evaluated, the good of the institution and its community, and the innate rights of individuals Ethical skepticism An ethical approach that says concrete and inviolate moral codes cannot be formulated Ethics The principles and guidelines that help us to uphold the things we value; A philosophic principle concerned with opinions about appropriate and inappropriate moral conduct or behavior by an individual or social group; Process by which we clarify right and wrong and act on what we take to be right; Code of practice imposed upon practitioners by their professional body and/or employer Ethics committee A formally constituted panel which conducts required reviews of the ethical grounds of research proposals prior to their approval Ethnocentrism Judging people from a different culture according to the standards and values of your own culture; The belief of the members of a group that their culture has the best beliefs, values and norms Ethnography Research design in which the researcher studies an intact cultural group in a natural setting during a prolonged period of time by collecting, primarily, observational data; The research process is flexible and typically evolves contextually in response to the lived realities encountered in the field setting; A form of qualitative research focused on describing the culture of a group of people; A type of qualitative research which treats a group of people as an anthropologist would an unknown tribe using detailed descriptions of how they live and act Ethnology The comparative study of cultural groups Ethogenic study

Assessment Glossary


59

An ongoing observational approach that focuses on processes rather than products Etic perspective An external, social scientific view of reality Etic term Outsider’s words or special words that are used by social scientists Evaluation A study to determine the extent to which a program or project reached its goals; Judgment regarding the quality, value, or worth of a response, product, or performance based on established criteria; An assessment at a point in time of the value, worth or impact of a project or program; Qualitative and quantitative descriptions of pupil behavior plus value judgments concerning the desirability of that behavior; Using collected information to make informed decisions about continued instruction, programs, activities; To measure, compare, and judge the quality of student work, schools, or a specific educational program; Using assessment findings (evidence or data) to judge program effectiveness; Using assessment findings as a basis for making decisions about program changes or improvement; Analyzing results and drawing conclusions; Decisions made about assessment findings; Deciding about the value of programs/program outcomes; Systematic investigation of the worth or merit of a student's performance in relation to a set of learner expectations or standards of performance; Systematic process of determining the merit, value, and worth of someone or something; Any effort to use evidence to improve effectiveness; May involve recommendations for changes; May utilize notable events and subjective impressions in addition to various measurements; The purposes of evaluation include finding out if client or community needs were met, improving a project or program, assessing the outcomes or impacts of a program, finding out how a program is operating, assessing the efficiency or cost-effectiveness of a program, and/or understanding why a program does or does not work Evaluation apprehension Type of confounding where naturally apprehensive or secretive personalities will not behave normally during a test; Potential bias indicated by comments such as "I better watch what I say in front of you”; Potential bias created when instructors do not trust administrators or the assessment process and believe or fear that the results of student learning assessments will be used to evaluate their own performance; May result in the confounding of standardized tests when instructors “teach to the test” Evaluator Anyone who accepts and executes responsibility for planning, conducting, and reporting evaluations; A person who assembles data and information collected about a subject, analyzes them, makes judgments as to whether that subject's performance level meets the pre-specified standards, prepares a summary report, writes recommendations, and who may provide feedback to the subject, directly or through another person Event sampling Observing only after specific events have occurred Evidence Proof, data, or documentation that fulfills a purpose, supports a conclusion or tests a hypothesis; Quantitative and qualitative data which an institution as a whole uses to determine the extent to which it attains the performance goals it establishes for itself Evidence-based practice (EBP) Properly informed professional decision making; The conscientious, explicit, and judicious use of current best evidence in making decisions; A systematic approach to integrating current scientific evidence; Only as good as the evidence available at the time and the process used to identify and select appropriate evidence Evidential basis of validity

Assessment Glossary


60

The assemblage of information about the construct validity of the test scores and measurements, as well as the relevance of the measurement to its applied purpose and its utility in an applied setting; Construct validity, in this case, includes information collected from content and criterion-related validation procedures; Ex post facto research After the fact or retrospective research; Studies investigating possible cause-and-effect relationships by observing an existing condition or state of affairs and searching back in time for plausible causal factors; A method of identifying the possible antecedents of an event that has already happened and which, thus, cannot be engineered or manipulated by the researcher; Research where the independent variable or variables have already occurred and in which the researcher starts with the observation of a dependent variable or variables; One of the recognized subtypes of the experimental method Excellence A quality or state of high or superior performance, or of having virtues and values surpassing most others Excess capacity The ability to produce additional units of output without increasing fixed capacity Executive report An abbreviated report that has been tailored specifically to address the concerns and questions of a person whose function is to administer a program, department, project, or institution Executive summary A brief, non technical summary statement designed to provide a quick overview of the full-length report on which it is based Exhaustive A set of categories that classify all of the relevant cases in the data; Property that the response categories include all possible responses; Sets covering the complete range of data values Exhibition An inclusive demonstration of skills or competencies; May be live performances, interdisciplinary, and require individual creativity as well as a display of developed skills; May include individual or group/collaborative projects produced over an extended period of time Existence versus frequency A key question in the coding process of content analysis; The researcher must decide if he/she is going to count a concept only once (for existence) no matter how many times it appears, or if he/she will count it each time it occurs (for frequency) Exit interviews or surveys Information obtained from students on completion of their study or leaving the institution; Includes information about student growth and change, satisfaction with academic programs, their experiences in their majors, reasons for leaving (if non completer), and their immediate and future plans Exit level Information obtained about students on completion of their study to assess the changes in learning as a result of their program of study Exit outcomes or proficiencies What all students should know and be able to do when they graduate from school Expectancy data Data illustrating the number or percentage of people that fall into various categories on a criterion measure

Assessment Glossary


61

Expectations Realistic goals or outcomes Expected growth The average amount of change in test scores that occurs over a specified time interval for individuals with certain individual characteristics such as age or grade level Expected outcome A specific and desired benefit that occurs to participants of a program following application of an intentional action; Generally phrased in terms of the changes in knowledge, skills, attitudes, behavior, condition, or status that are expected to occur in the participants as a result of implementing the action Expensing The depreciation of the entire cost of an asset during the first year of purchase Experience That which is acquired through exposure to, or participation in, an activity or process Experience curve Systematic cost reductions that occur over the life of a product; Product costs typically decline by a specific amount each time accumulated output is doubled Experiment An environment in which the researcher objectively observes phenomena that are made to occur in a strictly controlled situation in which one or more variables are varied and the others are kept constant; A researcher creates an environment in which to observe and interpret the results of a research question, participants in a study are randomly assigned to groups, and in an attempt to create a causal model groups are treated differently and measurements are conducted to determine if different treatments appear to lead to different effects; Performance of a planned set of trials; Any process or study which results in the collection of data, the outcome of which is unknown; In statistics, the term is usually restricted to situations in which the researcher has control over some of the conditions under which the experiment takes place Experimental control Eliminating any differential influence of extraneous variables Experimental design The art of planning and executing experiments; The plan of an experiment, including selection of subjects, order of administration of the experimental treatment, the kind of treatment, the procedures by which it is administered, and the recording of the data (with special reference to the particular statistical analyses to be performed); Refers to the way that experiments or studies are structured to provide meaningful data relevant to the research questions being investigated Experimental group or sampling unit The group that receives the experimental treatment condition; A unit is a person, animal, plant or thing which is actually studied by a researcher; The basic objects upon which the study or experiment is carried out; The entities to which the treatments are applied; A group of subjects assigned to receive a treatment (the independent variable), the effects of which (the dependent variable) are measured and compared to those observed for a control (comparison or non treatment) group; The group of subjects who receive the experimental treatment or process which is under investigation Experimental methods

Assessment Glossary


62

Class of research design intended to approximate to the ideal of the true experiment and characterized by structured observation of the effects of one or more deliberately manipulated independent variables on a single dependent variable, while the effects of other (ideally all) possible causation is tightly controlled Experimental research Research in which the researcher manipulates the independent variable; Scientific investigation in which an investigator manipulates and controls one or more independent variables to determine their effects on the outcome or dependent variable; Research which seeks explanations for cause-effect relationships by manipulating one or more factors (independent variables) while holding other factors constant Experimenter or experimenter bias effect The unintentional effect that the researcher can have on the outcome of a study; Consequence of carelessness and lack of attention to detail by experimenters which bias the results of their research (e.g. by failing to prevent confounding or not enforcing experimental controls) Experimenter expectancy Type of confounding where the expectations of experimenters subtly influence the behaviors being measured Expert One who has demonstrated a high level of proficiency in a knowledge area or skill set; An individual who has demonstrated expert level performance on an assessment; Also used to describe an individual with sufficient knowledge in an area to evaluate the content relevance and content representativeness of a measure and/or to help set standards in an area Explanation Attempting to show how and why a phenomenon operates as it does Explanatory research Testing hypotheses and theories that explain how and why a phenomenon operates as it does Exploratory factor analysis An analysis of latent factor structures conducted without a priori constraints or hypotheses Exploration Attempting to generate ideas about phenomena Extant data Existing data, information, and observations that have been collected or that are available for use in the assessment and evaluation processes External assessment or evaluation Evaluation conducted by an evaluator from outside the organization within which the object of the study is housed; Use of criteria or instruments developed by individuals or organizations external to the one being assessed; Use of evaluators not employed by the assessed institution; Usually summative, quantitative, and often high stakes; Sometimes used to refer to an assessment of external factors (i.e. an environmental scan) External audit An assessment conducted by an independent external auditor as a means of providing a professional opinion on the accuracy and fairness of a company’s financial statements based on generally accepted auditing procedures; A systematic assessment of a organization’s management, finances, operations, controls, and scope in which policies and procedures are carried out, performed by professionals who are independent of the entity being audited; The periodic external scrutiny of corporate governance and management systems, financial statements, and underlying financial systems, and the performance, performance management, and reporting of

Assessment Glossary


63

public bodies; Periodic or specific purpose (ad hoc) audit conducted by external (independent) qualified professionals External auditor An audit professional who performs an audit on the financial statements of a company, government, individual, or any other legal entity or organization, and who is independent of the entity being audited External benchmarking The process of comparing an organization’s current policies and practices to those of a competitor organization(s) to determine current and future trends in areas of employment and business practice External criticism Determining the validity, trustworthiness, or authenticity of the source External examiner Using an expert in the field from outside your program, usually from a similar program at another institution to conduct, evaluate, or supplement assessment of your students External reliability The degree of consistency of a measure over time; See also Internal reliability External validity The extent to which the study results can be generalized to and across populations of persons, settings, times, outcomes, and treatment variations; The extent to which the results of a study can be generalized or transferred to another setting; The extent to which assessment findings and conclusions from a study conducted on a sample population can be applied to the population at large; The ability to apply the findings in one context to another similar context

Externalities A cost or benefit imposed on one party by the actions of another party; Costs are negative externalities and benefits are positive externalities Extraneous variable A variable that may compete with the independent variable in explaining the outcome; any variable other than the independent variable that might influence the dependent variable Extrapolate or extrapolation To infer an unknown from something that is known; To estimate the value of a variable outside its observed range; To predict future results from prior observed results Extreme case sampling Identifying the “extremes” or poles of some characteristic and then selecting cases representing these extremes for examination

Assessment Glossary


64

F Face validity The perceived extent of acceptability or legitimacy of an instrument or process to teachers, administrators, policymakers, students, parents, the general public, and other stakeholders concerned with educational evaluation and the quality of learning/teaching; Extent to which an instrument superficially appears to an examinee as measuring the construct being investigated; The degree to which a measure appears to the subject being measured as measuring what it claims to measure; Face validity may motivate subjects to perform well or it may promote deliberate misrepresentation by a respondent; Apparent or presumed validity which has not been tested or validated; An evaluation of a test based on inspection only; Making a decision regarding the appropriateness of a test or other assessment instrument based on appearance rather than objective criteria Facesheet codes Codes that apply to a complete document or case Facilitation Coordinating rather than leading an exercise so that all group members are encouraged to participate in the discussion or activity; Helping a group of people come to conclusions Facilitator A person who makes it easier for other people to accomplish objectives by offering advice and assistance in solving problems, either with technical issues or with other people Factor Input variable; Any variable, real or hypothetical, that is an aspect of a concept or construct; In measurement theory, a statistical dimension defined by factor analysis; In mental measurement, a hypothetical trait, ability, or component of ability that underlies and influences performance on two or more tests and hence causes scores on tests to be correlated; In analysis of variance, an independent variable; In factor analysis, a cluster of related variables that are distinguishable components of a larger set of variables; A number by which another number is multiplied, as in “gas prices tripled or increased by a factor or 3 within five years” Factor analysis A statistical procedure that analyzes correlations among test items and tells you the number of factors present, and whether the test is unidimensional or multidimensional; A statistical test that explores which variables in a data set are most related to each other; A survey factor analysis can yield information on patterns of responses, not simply data on a single response; Allows larger tendencies to be interpreted, indicating behavior trends rather than simply responses to specific questions; Any of several statistical methods of describing the interrelationships of a set of variables by statistically deriving new variables, called factors, that are fewer in number than the original set of variables; Factor analysis reveals how much of the variation in each of the original measures arises from, or is associated with, each of the hypothetical factors; Any of several methods used for reducing correlated data to a smaller number of factors where a small number of factors considered to be the basic variables accounting for interrelations in the data are extracted from a correlation matrix; Method requiring the accumulation of scores on a number of simultaneous variables for each subject and then performing multiple correlations; See also Principal components analysis Factorial design A design in which two or more independent variables, at least one of which is manipulated, are simultaneously studied to determine their independent and interactive effects on the dependent variable; Participants must be randomly assigned to the levels of at least one of the independent variables Factorial design based on a mixed model

Assessment Glossary


65

A factorial design in which different participants are randomly assigned to the different levels of one independent variable but all participants take all levels of another independent variable Fair or fairness Impartiality; Such aspects of the assessment program and evaluation system as equal opportunity to acquire the knowledge and skills to be assessed, use of developmentally appropriate assessments, sound procedures, appropriate use of evaluation results, and reasonable demands on the subjects being evaluated in terms of such factors as time, costs, and personal resources required; Assessment or test that provides an even playing field for all students; Teachers, students, parents and administrators agree that the instrument has validity, reliability, and authenticity, and they therefore have confidence in the instrument and its results False negative When assessment results inaccurately indicate a person, program, or organization should not be placed into a category; An incorrect diagnostic judgment that an entity does not fall within a target category False positive When assessment results inaccurately indicate a person, program, or organization should be placed into a category; An incorrect diagnostic judgment that an entity does fall within a target category Fat organization An organization with a structure consisting of several layers of management Fatigue effect A type of confounding possible with prolonged or physically demanding research procedures resulting in reduced or poorer performance on later items in a procedure or instrument FERPA Family Educational Rights and Privacy Act of 1974 Feasibility The extent to which an evaluation is appropriate and practical for implementation; Capability of being accomplished or brought about Feasibility study A study designed to discover if a business, product, project, or process justify the investment of time, money, and other resources Federal mandates Any provision in a bill or joint resolution before Congress or in a proposed or final Federal regulation that would impose a duty that is enforceable by administrative, civil, or criminal penalty or by injunction (other than a condition of the bill or joint resolution or implementing regulation) Feedback Information that provides the performer with direct, usable insights into current performance, based on tangible differences between current performance and hoped-for performance; The information and recommendations provided to a learner/teacher about his/her performance based on the results of that learner/teacher's evaluation and designed to help the learner/teacher improve his/her performance and make decisions concerning professional development and improvement Feedback forms or feedback questionnaires Forms that gather information about participants and their assessments of an activity, course or program Field notes

Assessment Glossary


66

Notes taken by an observer; Notes taken by researchers to record unstructured observations they make in the field and their interpretation of those observations Field test Studies of a program, project, or instructional material in settings like those where it is to be used Financial aid Funds awarded to a student on the basis of his/her demonstrated financial need for the purpose of meeting postsecondary education expenses Financial assistance Any support received by a student from a source other than parents, spouse, or his/her own resources, to help meet the student's postsecondary education expenses Financial audit A review of a company’s financial position serving as confirmation that a company’s financial statements are accurate Financial budget A budget that contains a company’s balance sheet detailing how a particular operations’ plan will affect company resources and liabilities Financial need The difference between the cost of a student's postsecondary education and the expected family contribution from the student and his/her family to be applied toward the student's postsecondary education expenses Financial statement A report containing financial information derived from an organizational accounting record First impression effect A rater effect in which an assessor or an evaluator tends to base judgments and evaluations on early opinions rather than on a complete picture and tends to distort subsequent information to fit the initial opinion First-in, first-out (FIFO) An inventory valuation technique that presumes inventory purchased first is used or sold first First mover advantage The competitive advantage held by a firm from being first in a market or first to use a particular strategy Fiscal Related to finance or finances Fixed assets An accounting term used to describe tangible property used in the day-to-day operation of a business; Typically includes items such as real estate, equipment, machinery, fixtures, and furnishings Fixed budget A type of budget that is developed based solely on fixed costs and does not alter with fluctuations in activity Fixed costs The day-to-day costs of doing business that don’t fluctuate based on the number of goods produced or services performed

Assessment Glossary


67

Flexible cognition A learning framework that encourages higher-order mental skills; The ability to spontaneously restructure one’s knowledge, in many ways, by adapting to the demands of radically changing situations Flat organization An organization characterized by having only a few layers of management from top to bottom Floor The lowest limit of performance that can be assessed or measured by an instrument or process; The lowest limit of performance that can be measured effectively by a test; Individuals have reached the floor of a test when they perform at the bottom of the range in which the test can make reliable discriminations; Individuals who perform near to or below this lower limit are said to have reached the floor, and the assessment may not be providing a valid estimate of their performance levels; Such individuals should be given a less difficult assessment of the same attribute if it is necessary to differentiate between lower levels of performance Focus group analysis or discussion A qualitative research process designed to elicit opinions, attitudes, beliefs, and perceptions from individuals to gain insights and information about a specific topic; A form of structured group discussion involving people with knowledge and interest in a particular topic and a facilitator; A carefully planned discussion to obtain perceptions on a defined area of interest in a permissive, non threatening environment; A group selected for its relevance to an evaluation that is engaged by a trained facilitator in a series of discussions designed for sharing insights, ideas, and observations on a topic of concern; A common type of group discussion, in which a moderator encourages a small group of people (usually 8 to 10) to gradually focus on a topic; Typically a group of 7-12 individuals who share certain characteristics relative to a particular topic related to a research or evaluation question in which discussions are conducted by a trained moderator to identify trends/patterns in perceptions; Moderators provide direction and set the tone for the group discussion, encourage active participation from all group members, and manage time; Moderators must not allow their own biases to be expressed verbally or nonverbally; Careful and systematic analysis of the discussions provides information that can be used to evaluate and/or improve the desired outcome; A small group of people, led by a facilitator, discuss their ideas about a particular issue Follow up Actions taken to maintain the strengths and address the weaknesses that were identified in the evaluation Forced choice response or assessment A format for a test, assessment, rating, or survey item where the respondent is given a limited number of options from which to select an answer; In cases where there is a correct or best answer, the other options are referred to as distractors; Testing where responses to an item, questions or prompts are placed against a set answer key; Scoring does not require judgment on the part of the scorer because there is one right answer to the item; In test construction, forced choice or multiple-choice items require an examinee to choose an answer from a small set of response options; Examples of forced-choice items are multiple-choice questions, true/false items, checklists, and a five-point rating scale Forecasting A business analysis conducted in order to assess what future trends are likely to happen, especially in connection with a particular situation, function, practice or process that is likely to affect the organization’s business operations Formal The conducting of an assessment or evaluation activity in accordance with a prescribed plan, structure, or advance notice Formal award

Assessment Glossary


68

A diploma, certificate, or degree awarded in recognition of the successful completion of a program of studies; Also included under this heading are certificates awarded for attendance or for the completion of a course Format The structure of assessment instruments, evaluation forms, and materials; Includes shape, size, mode of delivery (e.g. paper versus computer, in-person versus telephone, audio tape versus video tape), and general design or layout of the materials and equipment Formative assessment or evaluation Improvement-oriented assessment; Evaluation conducted while a creative process is under way, designed and used to promote growth and improvement in a student's performance or in a program's development; Evaluation designed and used to improve an object, especially when it is still being developed; Process of judging an ongoing, changing process or product for diagnosis, revision, description, information, or comparison; Observations which allow one to determine the degree to which students know or are able to do a given learning task, which identifies the part of the task that the student does not know or is unable to do, and which suggests future steps for teaching and learning; Ongoing, diagnostic assessment providing information, or feedback, to guide instruction and improve student performance; The gathering of information about student learning during the progression of a course or program; Improvement-oriented assessment that usually takes place continually throughout a lesson module, course, or program; The use of a broad range of instruments and procedures during a course of instruction or during a period of organizational operations in order to facilitate mid-course adjustments; Usually conducted repeatedly to tract improvement or change in the learning of students over time; May be short term focused to assess whether some or all students in a group need additional assistance to achieve a course outcome; May be long term focused to assure students progress through their courses toward achieving program outcomes; Formative research (often called formative evaluation) is conducted during the planning and delivery of a program and it produces information that is used to improve a program while it is in progress; Formative research can be contrasted to summative research, which assesses a program upon its completion Fraudulent activity Fabrication or alteration of results Frequency analysis Frequency analysis measures the number of times a particular distracter, or combination of distracters, was selected by a group of participants Frequency distribution Arrangement in which the frequencies of each unique data value are shown; A statistical description of raw data in terms of the number of items characterized by each of a series of values of a continuous variable Frontload Amount of effort required in the early stage of assessment method development or data collection Frontstage behavior What people want or allow us to see Full costs A cost management and cost-benefit analysis of the sum of direct and indirect costs; Costs and advantages may be considered in terms of environmental, economical, and social impacts Fully anchored rating scale All points are anchored on the rating scale Functional-level testing

Assessment Glossary


69

The administration of an instrument or assessment process whose difficulty level is appropriate for the individuals being tested, but not necessarily for the age or grade level or group membership of that individual; Functional-level testing refers to the individual being tested, not to the group for whom the assessment was designed nor the group to which the individual belongs Fundamental principle of mixed research Advises researchers to mix research methods or procedures in a way that the resulting mixture or combination has complementary strengths and non overlapping weaknesses; The researcher should use a mixture or combination of methods that has complementary strengths and non overlapping weaknesses

Assessment Glossary


70

G Gain scores The difference between a student's performance on a test and his or her performance on a previous administration of the same or parallel test Gantt chart A common way of showing tasks over time used for project planning and keeping track of the status of individual tasks within a project; A horizontal bar chart developed by Henry L. Gantt as a production control tool; Used in project management to provide a graphical illustration of a schedule that helps to plan, coordinate and track specific tasks in a project; A graphical time-line which represents the durations of multiple tasks or activities used in planning and coordination; Gantt charts are constructed with a horizontal axis representing the total time span of the project broken down into increments (days, weeks, or months), a vertical axis representing the tasks that make up the project, and horizontal bars for each task connecting the starting and ending times; Gantt charts often include milestones (important checkpoints or interim goals), delegated responsibilities (who does what when), current completion status (with charts updated by filling in task bars to a length proportional to the amount of work that has been finished), and dependencies (activities which are dependent on the prior completion of other activities) or other relationships between tasks Gap analysis A means of measuring and evaluating differences between an organization’s current position and its desired future position; The identification of the difference between the desired and current state Garbage can theory of organizational choice A theory of organizational decision making applicable to organizations where goals are unclear, technologies are imperfectly understood, histories are difficult to interpret, and participants wander in and out; Such "organized anarchies" operate under conditions of pervasive ambiguity, with so much uncertainty in the decision making process that traditional theories about coping with uncertainty do not apply General education The common knowledge, skills, and developmental characteristics of the college educated person General linear model (GLM) The mathematical model underlying most comparative statistics, involving proportional relationships among variables of interest; A mathematical procedure that is the “parent” of many statistical techniques General obligation bond A bond that guarantees that all taxpayers will be responsible for the bond’s principal and interest payments Generalizable or generalizability Results of an assessment are generalizable when the score on one assessment can accurately predict a student score on a different assessment covering the same knowledge or skill; Generalizability across time is promoted by ensuring that assessments focus on general level concepts or strategies, not on facts, topics, or skills, which are found only at one level or in one class; The extent to which research findings and conclusions from a study conducted on a sample population can be applied to the population at large; The extent to which the performances sampled by a set of assessment activities are representative of the broader domain being assessed; The extent to which information about a program, project, or instructional material collected in one setting can be used to reach a valid judgment about how it will perform in other settings; The appropriateness of using results from one context or purpose in another context or for another purpose Generalizability coefficient

Assessment Glossary


71

A reliability index encompassing one or more independent sources of error; Formed as the ratio of the sum of variances that are considered components of test score variance in the setting under study to the foregoing sum, plus the weighted sum of variances attributable to various error sources in this setting; Indices which arise from the application of generalizability theory are typically interpreted in the same manner as reliability coefficients Generalizability study (G-Study) A type of investigation that applies generalizability measurement theory; Provides researchers with an indication of the extent to which a sample of measurements generalizes to a universe of measurements Generalizability theory (G-Theory) An expansion of classical test theory (CTT) that uses variance components to create coefficients, parallel to reliability coefficients in CTT that informs researchers about the generalizability of measures Generalize Making statements about a population based on sample data Generative skills Skills or knowledge that are durable over time in face of changes in technology, work process, and societal demands: They cross functions and serve as the foundation for effective adaptation to changes in role requirements GIGO Short for "garbage in, garbage out", indicating that your results are only as good as your input Goal articulation A process of defining and clearly expressing goals generally held by those in an organization or group; Usually regarded as a function of organization or group leaders and a key step in developing support for official goals Goal congruence Agreement on fundamental goals in the context of an organization; Refers to agreement among leaders and followers in the organization on central objectives; In practice, its absence in many instances creates internal tension and difficulties in goal definition Goal free evaluation Evaluation of outcomes in which the evaluator functions without knowledge of the purposes or goals; Focuses on actual outcomes rather than intended program outcomes Goals Desired end results; A statement of intent or an end that a person or a group strives to attain; Statements of expectations of general capabilities or student outcomes resulting from planned educational experiences; Intended results stated in general terms; A written statement describing the direction that indicates success or improvement in organizational performance; Broader and less specific statements than objectives or outcomes; Derived from or aligned with the mission statement of the institution and department or unit; In terms of student behavior, delineates a less focused outcome than an objective; Instructional goals: General or global statements about knowledge, skills, attitudes, and values expected of students completing a course, discipline, or program; Non-instructional goals: General or global statements about level of productivity, quality or effectiveness of services provided by non-instructional units; Goals for Learning: Expressions of intended results in general terms used to describe broad learning concepts or competencies (such as clear communication, problem solving, information literacy, and ethical awareness) Going native Identifying so completely with the group being studied that you can no longer remain objective

Assessment Glossary


72

Governance A separate management system or leadership process used to lead, direct, command, and control an organization Grade or mark A rating or evaluation of a student's achievement in a course Grade or grade level equivalent A score that describes student performance in terms of the statistical performance of an average student at a given grade level; The estimated grade level that corresponds to a given score; School grade level for a given population for which a given score is the median score in that population; A grade equivalent score does not equate to performance in the classroom Grade or grade level norms The average test score obtained by students classified at a given grade placement; Interpreting scores on a test in reference to the average performance of students at each grade level Grade-point average (GPA) A measure of a student's average performance in all courses taken during a reporting period, term, or academic year—or accumulated for several terms or years—except for those courses explicitly excluded by the institution from grade-point average calculations; This numerical average is found by (1) multiplying the numerical grade or a number assigned to a non numerical grade (for each course to be included in average), by the number of award units (for each course in the average), (2) summing these results, and (3) dividing this sum by the total award units for all courses included; Grade points for a course are obtained by multiplying the number of award units given for successful course completion by the numerical grade (or equivalents for non numerical grades) awarded for student performance (e.g. A =4, B = 3, C = 2, D = 1, and F = 0) Grading Values placed on varying levels of achievement in course assignments and/or examinations; Graded course assignments and tests can be used for assessment if they are clearly linked to course or program goals and if the assessment is based on established criteria and standards Grading system The process by which a teacher arrives at the symbol, number, or narrative presentation that is used to represent a student's achievement in a content or learning area Graduation rate Extent to which students at an institution of postsecondary education complete an academic program, or enroll in a subsequent educational program for which the prior program provided substantial preparation, within 150 percent of the normal time for completion of the program Grant A giving of funds for a specific purpose Graphic organizers Mental maps that help students make their thinking visible; Representations of the process skills of sequencing, comparing, contrasting, classifying, inferring, drawing conclusions, problem solving, and thinking critically Grounded theory A qualitative approach to generating and developing a theory from the data that the researcher collects; A general methodology for developing theory that is grounded in data systematically gathered and analyzed; Theory derived from continuously testing an assumption so that it evolves over time in response to collected information until it is no longer an assumption; Practice of developing other theories that emerge from observing a group in which the theories are grounded in the group's observable experiences, but researchers add their

Assessment Glossary


73

own insight into why those experiences exist; Researcher attempts to derive a theory by using multiple stages of data collection and the refinement and interrelationship of categories of information; Two primary characteristics of this design are the constant comparison data with emerging categories, and theoretical sampling of different groups to maximize the similarities and the differences of information; An approach to qualitative research where the researchers try to approach a problem with no preconceptions and to build their theories solely from the data gathered Group affiliation Membership association; Civic development or citizenship Group discussion A generic type of qualitative research in which a small group of people provide information by discussing a topic Group moderator The person leading the focus group discussion Group think A tendency of individuals to adopt the perspective of the group as a whole which tend to occurs when decision makers don’t question underlying assumptions Grouped frequency distribution Distribution where the data values are clustered or grouped into separate intervals and the frequencies of each interval are given Growth over time Increases in knowledge or development that occur over time, typically measured through pretest/posttest experimental designs Guessing parameter The probability that a student with very low ability on the trait being measured will answer the item correctly Guideline A procedural suggestion intended to help evaluators and their audiences to meet the requirements of the evaluation standards; Strategy to avoid mistakes in applying the standards Guiding principles Underlying tenets or assumptions that describe effective learning, teaching and assessment within each subject area Guttman scale A type of response scale where items assessing a unidimensional construct are ordered by difficulty (using classical test theory); When an examinee fails to endorse an item it is expected that he/she will not endorse any harder item, and when an examinee endorses an item it is expected that he/she will endorse all easier items; This type of scale was a predecessor to item response theory

Assessment Glossary


74

H Habits of mind A fluid and life-long approach to learning that involves reflection, inquiry, and action; An approach that favors uncovering concepts rather than covering content and encourages the learner to think about how they acquire knowledge Halo effect A type of rater effect in which an assessor or an evaluator tends to base overall judgments or evaluations on selected pieces of information rather than on all available relevant information Hawthorne effect Phenomenon whereby behavior can be changed by the mere act of observing it; Changes in outcomes are the result of the experimental group receiving a treatment but are not a result of the specific treatment received; Similar to the placebo effect in medical research; A form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied rather than in response to any particular experimental manipulation; See Attention bias Hidden curriculum Beliefs, values and understandings that are passed on to the pupil or student, not explicitly, but unconsciously and implicitly through the organization of the institution Hierarchy A characteristic of formal bureaucratic organizations; A clear vertical "chain of command" in which each unit is subordinate to the one above it and superior to the one below it; One of the most common features of governmental and other bureaucratic organizations High inference The types of judgments, decisions, and conclusions that are based on complex inductive reasoning and that require a high degree of subjectivity on the part of the individual High performance organization Systems promoting a culture of learning and improvement High stakes assessment or evaluation Evaluations that lead to decisions that, if incorrect, harm students and are detrimental to their future progress and development; Any testing program whose results have important consequences for students, teachers, schools, and/or districts, such as promotion, certification, graduation, or denial/approval of services and opportunity; When assessment results are tied to student graduation or to funding for schools from states or other agencies; Using the results of an assessment to set a hurdle that must to be cleared for completing a program of study, receiving certification, or moving to the next level; Usually such assessments are externally developed, based on set standards, carried out in a secure testing situation, and administered at a single point in time; Examples include mandated exit exams required for graduation and bar exams required for practicing law in a specific jurisdiction; Can corrupt the evaluation process when pressure to produce rising test scores results in "teaching to the test" or making tests less complex High stakes test A test whose results have important, direct consequences for the examinees, programs, or institutions tested Higher order cognitive development

Assessment Glossary


75

Defined as the development of understanding one's own and others' knowing and serves as the supporting structure for sustained intellectual inquiry and analysis Higher order thinking A complex level of thinking that entails analyzing and classifying or organizing perceived qualities or relationships, meaningfully combining concepts and principles verbally or in the production of art works or performances, and then synthesizing ideas into supportable, encompassing thoughts or generalizations that hold true for many situations Holistic rubric A holistic rubric assigns a level of performance by assessing performance across multiple criteria as a whole instead of listing separate levels of performance for each criterion Holistic perspective Taking almost every action or communication of the whole phenomenon of a certain community or culture into account in research Holistic scoring or holistic method Method of scoring essays, products, and performances in which a single score is given to represent the overall quality of the essay, product, or performance without reference to particular dimensions; Assigning a single score based on an overall assessment of performance rather than by scoring or analyzing dimensions individually; Used when the product is considered to be more than the sum of its parts and so the quality of a final product or performance is evaluated rather than the process or dimension of performance; Scoring procedure yielding a single score based upon an overall impression of a product or performance; Evaluation score is based on an overall impression of student performance rather than multiple dimensions of performance; Type of grading in which an assignment is given an overall score with possible scores described in a rating scale; In a holistic rating scale a high score indicates achievement of all aspects of the assignment, while a low score means few if any of the desired outcomes have been achieved; Holistic score levels need to be specific enough to reveal meaningful, diagnostic information when the scores are aggregated Horizontal organization A flat organizational structure that consists of fewer hierarchal levels and which often relies on the use of cross-functional teams Human capital The collective knowledge, skills, and abilities of an organization’s employees Human relations theories of organization Theories stressing workers' noneconomic needs and motivations on the job which seek to identify those needs and how to satisfy them; Organizational theories which focus on working conditions and social interactions among workers Human resources planning The process of anticipating future staffing needs and ensuring that a sufficient pool of talent possessing the skills and experience needed will be available to meet those needs Hybrid organization An organization whose structure is comprised of both vertical and horizontal models Hypothesis A tentative explanation based on theory to predict a causal relationship between variables; Statement of the casual relationship between two (or more) variables (e.g. “if X, then Y” or “if variable A is X, then variable B will be Y”)

Assessment Glossary


76

Hypothesis testing The act of applying objective and peer-replicable testing to theoretical beliefs; Normally supported by using inferential statistics in attempts to disprove the null hypothesis and provide an estimate of the confidence level in the form of a p-value Hypothetical construct A presumed internal quality of a system, beyond direct observation, whose presumed operation accords with available empirical data; A construct is some postulated attribute of people, assumed to be reflected in test performance; Constructs are part of a theory and may, in turn, map onto one or more variables, each of which may be operationalized as observable measures in a number of different ways; See Construct

Assessment Glossary


77

I I-E-O model Astin's Input-Environment-Outcome model of student development; This model describes the development of students as being an interaction of the college environment and the characteristics students bring with them to the environment Identity statement A concise statement of salient and distinguishing characteristics of an institution (who we are and how we are different) Impact data Summary data that describe the likely results of a decision (e.g. number of students likely to fail to meet a given cut score) Impact evaluation Evaluations that look specifically at whether or not the program achieved its goals and had its intended effect; An evaluation measures the final results of a program or initiative; See also Outcome evaluation Impacts Changes that occur in the family, community, and larger society as a consequence of participation in adult literacy education Implementation Putting something into place by doing what was planned Improvement Making something better by moving closer to a goal In-house instruments or software Non-proprietary instruments/software which are tools developed by institutions for internal use, not researched, or purchased from an outside source; In-house assessment tools are sometimes preferred because they are designed to exactly match an institutional purpose In-person interview An interview conducted face-to-face Inclusive approach Planning or decision-making which involves people and groups from diverse backgrounds or with differing interests that enables and encourages all participants in a project, process, or organization to be listened to and heard; Inclusive methods respect and recognize the differences between people, taking different views into account; Involves creating a relaxed and enjoyable atmosphere that allows ample time for participation, avoids the use of jargon or discriminatory language, and enables the development of trust and understanding between people or groups Income statement A financial document showing a company’s income and expenses over a given period of time Incompatibility thesis The proposition that one cannot mix quantitative and qualitative research

Assessment Glossary


78

Inconsistency A type of rater effect in which an assessor or an evaluator tends to rate or to interpret similar data and information in different ways Incrementalism A model of decision making that stresses making decisions through limited successive comparisons, in contrast to the rational model; Focuses on simplifying choices rather than aspiring to complete problem analyses, on "satisfying" rather than "maximizing" Incubator culture A corporate culture high on personal and egalitarian dimensions Independent variable A variable that is presumed to cause a change in another variable; A variable that is part of the situation that exist and from which originates the stimulus given to a dependent variable; A variable that is considered to be a cause or a variable that is used for prediction; The variable or factor that is assumed to cause or influence the dependent variable(s) or outcome; The independent variable is manipulated in experimental research to observe its effect on the dependent variable(s) Indicator or indicators Statistics that reflect something about the performance or status of some aspects of an educational program; A specific description of an outcome in terms of observable and assessable behaviors; Specifies what a person who possesses the qualities articulated in an outcome knows or can do; Measures for individuals or organizations that provide information about measurable traits, situations, knowledge, skills, performances, resources, inputs, or outputs; Specific examples and explicit definitions that can be used in rating students' level of achievement relative to specified skills, strategies, and knowledge; Several indicators are generally needed to adequately describe each outcome Indirect assessment Assessments that deduce student achievement of learning outcomes through the students’ reported perception of their own learning; May also be the opinions or thoughts of others about student knowledge, skills, attitudes, learning experiences, and perceptions Indirect costs Expenses incurred for the purpose of common general activities and cannot be identified or charged directly to a specific project, product, or service Indirect effect An effect occurring through an intervening variable Indirect labor Labor that is necessary to support product production or service delivery but is not directly involved with the actual production or delivery process Indirect evidence, methods, or measures Unobtrusive measures that occur naturally in a research context; Data collected by the researcher without introducing any formal measurement procedure; Evidence that shows student learning indirectly; Measures from which student learning can be inferred rather than directly demonstrated; Acquiring evidence about how students feel about learning and their learning environment rather than actual demonstrations of outcome achievement; Methods that involve perceptions of learning rather than actual demonstrations of outcome achievement; Students or others report their perception of how well a given learning outcome has been achieved; Quantitative examples include course grades, transfer rates and retention data; Qualitative examples include surveys of students, alumni, or employers, exit interviews, focus groups, and reflective essays;

Assessment Glossary


79

Processes employed to assess a program and/or student learning indirectly by gathering indirect information related to student learning or program success Indirect measures of learning See indirect evidence Individualism The cultural value dimension that emphasizes self-interest, independence, or self-reliance over group, social, or collective values, processes, or goals Inductive analysis A form of analysis based on inductive reasoning; A researcher using inductive analysis starts with answers, but forms questions throughout the research process Inductive codes Codes that are generated by a researcher by directly examining the data Inductive method A bottom-up or generative approach to research Inductive reasoning Reasoning from the particular to the general; A form of reasoning in which a generalized conclusion is formulated from particular instances; Drawing a hypothesis/assumption and collecting data to see if it is correct or not; Reasoning from the specific to the general Inclusivity Broadness in orientation or scope; Frequently used in terms of broadness of culture and ethnicity Industrial psychology Applied psychology concerned with the study of human behavior in the workplace and how to efficiently manage an industrial labor force and problems encountered by employees Inference A logical conclusion or judgment that is explicitly supported by data, evidence, and information gathered as part of the evaluation, assessment, or research process Inferential statistics The division of statistics focused on going beyond the immediate data and inferring the characteristics of a population based on samples; Research statistics; A measure of the confidence we can have in our descriptive statistics; The statistics we use to test hypotheses about a population; Inferential statistics is always guessing from incomplete information and the trick is being able to make definite statements regarding the accuracy of the guess; Class of mathematical procedures designed to establish the likelihood of a causal relationship existing between blocks of empirical observations Influence Attempting to apply research to make certain outcomes occur Informal The conducting of an assessment or evaluation activity without a prescribed plan or structure, or with little or no advance notice Informal test

Assessment Glossary


80

A non standardized test that is designed to give an approximate index of an individual's level of ability or learning style; often teacher-constructed Informal conversational interview Spontaneous, loosely structured interview Information Knowledge about the attributes and performance of a learner/teacher based on assessments, documentation, and data sources used in the assessment and evaluation processes Information competency, literacy or skills Combined capability to access, evaluate, and use information in fulfillment of coursework and independent study; The ability to acquire, evaluate, organize, maintain, interpret, and communicate knowledge Information needs Information requirements of the evaluator, clients, and other pertinent audiences to be met by the evaluation Information sources The persons, groups, instruments, recordings, and documents from which data are obtained Informed consent Agreeing to participate in a study after being informed of its purpose, procedures, risks, benefits, alternative procedures, and limits of confidentiality; Agreement by a subject, prior to the collection of information and/or its release in evaluation reports, that their names and/or other confidential information supplied by them may be used in specified ways, for stated purposes, and in light of possible consequences; The agreement between concerned parties about the data-gathering process and/or the disclosure, reporting, and/or use of data, information, and/or results from an assessment or evaluation; Agreement of a subject to participate in an experiment after having been informed of all the potential negative consequences of the experimental treatment; An individual’s agreement to allow something to transpire subsequent to the individual having been informed of associated risks and alternatives; The process of obtaining voluntary participation of individuals in research based on a full understanding of the possible benefits and risks of the research; Informed consent is usually demonstrated by signing a consent form Informed decision making Using data and information, not assumptions, to make decisions Infrastructure The fundamental physical structures of a city, county, state, country, or other geopolitical entity (includes roads, utilities, water, sewage, etc.) Input variables The activities, materials, and teacher behaviors designed to improve student learning and behavior Inputs The “nouns” of projects; The resources that are used to make the project happen (e.g. people and equipment); The personal, background, and educational characteristics that students bring with them to postsecondary education that can influence educational outcomes Institutional assessment Assessment to determine the extent to which a college or university is achieving its mission Institutional effectiveness (IE) The measure of what an institution actually achieves; How well the institution meets its goals and mission; How well an institution succeeds in accomplishing its mission as determined by the relative accomplishment of

Assessment Glossary


81

explicit objectives within a given time frame: An analysis of institutional effectiveness data measures (i.e., effectiveness, efficiency, and excellence) via outcomes assessment identifies opportunities for improvement; Extent to which an institution meets its unique mission; Usually includes the functions associated with assessment and institutional research; A measurement of institutional achievement done for the purpose of continual improvement; A continuous set of the processes which include planning, assessment, and review aimed at ongoing improvement; The ongoing quest for quality and the demonstration of how well the institution is fulfilling its mission and realizing its vision Institutional effectiveness plan The institution’s documented assessment plan that includes reporting plans of annual goals and objectives and follow-up assessment reports of designated departments; The written plan encompassing the college’s policies and procedures for assessment, and the reporting of assessment results Institutional outcome Benefits or changes in a population, organization, or community due to program activities and typically expressed in aggregate form; These benefits may include research, economic, civic, service, and community outcomes, as well as student learning outcomes Institutional performance The degree to which institutions and organizations of higher education have achieved their goals and objectives as evidenced by student outcomes and efficiency indicators Institutional portfolios A means of assessing the impact of the entire educational experience on student learning; May be used to drive internal improvement and external accountability Institutional Priorities Survey (I.P.) A proprietary survey by Noel-Levitz for Faculty and staff that measures the satisfaction and opinions of a wide range of support services at a college; Frequently used as a companion survey to the Student Satisfaction Inventory (SSI) to compare staff and student perceptions about an institution’s culture and environment Institutional research Applied study of a college's functioning in terms including measures of resources, inputs, and outputs and occasionally outcome assessment Instruction The methods and processes used by teachers to change what students know, what they can do, the attitudes they hold, or their behavior; Systematic provision of information, opportunities, and resources to promote the development of a repertoire of knowledge and skills Instructional assessment Determining the extent to which students have met learning outcomes (at the course, department, or program level) Instructional goal A statement of what students are expected to learn in a given lesson, unit, course, program, or across educational and training programs Instructional objectives More detailed expressions of educational objectives Instructional unit Any department directly related to students learning in the classroom

Assessment Glossary


82

Instructional validity The degree to which the items on a test measure what is actually being taught, and/or what the individuals being assessed have had an opportunity to learn Instrument An assessment device adopted, adapted, or constructed for the purposes of the evaluation; Research tool used to assess variables during an evaluation; Device used to collect data, information, and evidence; For example: Tests, questionnaires, application forms, interview schedules and protocals, checklists, rating scales, focus group protocols, and observation records Instrument or assessment tool A method of gathering data about student performance; For example: A questionnaire, a test, a checklist of stages in solving a problem, or a criterion-referenced rating scale for an art product or performance Instrumental case study Case study interested in understanding something more general than the particular case Instrumentation Any change that occurs in the way the dependent variable is measured Intangible assets or capital Assets or items of value which are not physical in nature (e.g. patents, trademarks, and brand recognition) Intellectual capital Broad term combining the idea of the intellect with the economic concept of capital referring to the collective knowledge of an organization applied to some money-making or other useful purpose Intellectual property Property protected under federal law which includes trade secrets, confidential or proprietary information, copyrightable or creative works, ideas, patents, or inventions Intelligence The ability to think abstractly and to learn readily from experience Intelligence test A psychological or educational test designed to measure intellectual processes in accord with some evidence-based theory of intelligence; A test that measures the higher intellectual capacities of a person, such as the ability to perceive and understand relationships and the ability to recall associated meaning; May measure the ability to learn Inter-coder reliability Consistency among different coders Inter-rater coefficients Special type of reliability coefficient used to determine the extent to which two or more raters are consistent in their scoring of students; Often used to determine whether two judges grade in the same way (e.g. would students receive the same grade if their responses were graded by two different teachers). Inter-rater or inter-observer reliability or agreement The degree to which different raters or observers give consistent estimates of the same phenomenon; Achieved when different coders arrive at the same results when the same body of material is examined; The extent to which two or more individuals agree which addresses the consistency of the implementation of a rating system Inter-scorer reliability

Assessment Glossary


83

The degree of agreement or consistency between two or more scorers, judges, or raters Interaction A measure of the extent to which the effect of one factor is different for different levels of another factor Interaction effect When the effect of one independent variable depends on the level of another independent variable Interdisciplinary The covering of knowledge and skills from several academic subject areas and/or domains Interdisciplinary or integrated assessment Assessing students’ abilities to apply concepts, principles, skills, and processes from two or more subject disciplines to a central question, theme, issue, or problem Intermediary A third party who serves as a facilitator for negotiating a deal or other transaction between parties Interim statement A report containing financial information covering a period of less than one year (i.e. quarterly reports) Internal audit The process of conducting an in-house examination of one or more of an organization’s processes, functions, programs, etc. Internal assessment Analysis of an organization’s position, performance, problems, and potential Internal consistency or reliability The consistency with which a test measures a single construct; The extent to which all questions or items assess the same characteristic, skill, or quality; A form of validation of a multiple item test where the criterion is the total score on the test itself; The extent to which a scale or instrument is measuring a single idea Internal consistency coefficient An index of the reliability of test scores derived from the statistical interrelationships of responses among item responses or scores on separate parts of a test; One of many types of reliability coefficients in classical test theory; A measure of the degree to which items on an instrument are inter-correlated Internal evaluation or audit An evaluation, assessment, or audit conducted by a staff member or unit from within the organization being studied Internal reliability See Internal consistency Internal reliability coefficient See Internal consistency coefficient Internal scanning Looking inside the business and identifying strengths and weaknesses of the firm Internal validity

Assessment Glossary


84

The ability to infer that a causal relationship exists between two variables; The rigor with which the study was conducted (e.g. the study's design, the care taken to conduct measurements, and decisions concerning what was and wasn't measured); Extent to which the designers of a study have taken into account alternative explanations for any causal relationships they explore Interpersonal communication Interpersonal communication occurs when two people engage in voluntary, ongoing, interdependent interactions, which involve meaningful interpretation of their verbal and nonverbal behaviors Interpersonal skills The personal ability to create unique shared meaning with others. Interpretation The act of explaining test scores to students so they understand exactly what each type of score means; For example: Explaining that a percentile rank refers to the percentage of students in the norm group who fall below a particular point, not the percentage of items answered correctly Interpretive validity Accurately portraying the meaning given by the participants to what is being studied Interrupted time-series design A design in which a treatment condition is assessed by comparing the pattern of pretest responses with the pattern of posttest responses obtained from a single group of participants Interval data A collection of observations of an interval variable Interval scale A scale of measurement that has equal intervals of distances between adjacent numbers Interval variable A variable in which both the order of data points and the distance between data points can be determined (e.g. percentage scores and distances) Intervening or mediating variable A variable that occurs between two other variables in a causal chain Intervention Doing something to change the possibility of the result; Actions taken to improve goal achievement; Changes made to improve outcomes or performance; The act of remediation itself, or the action taken to remediate a condition; The treatment which is actually delivered to the person; The alternative to doing nothing and letting nature take its course Intervention study Research designed to establish the efficacy and/or effectiveness of a remediation, treatment, program, etc Interview or interviewing Asking questions in a systematic way to determine another’s perspective or experiences; Asking individuals to share their perceptions of their own attitudes and/or behaviors or those of others; A face-to-face or long-distance (e.g. by telephone) discussion between two or more people to collect information and the opinions of the people being interviewed about certain topics or events; Evaluating student reports of their attitudes and/or behaviors in a face-to-face-dialogue; A survey research technique; Interviews are often audio- or video-taped for later transcription and analysis; A series of orally-delivered questions designed to elicit responses concerning attitudes, information, interests, knowledge, and opinions; May be conducted in person or by telephone, and

Assessment Glossary


85

with one subject or a group of subjects; The three major types of interviews are as follows: Structured: All questions to be asked by the interviewer are specified in advance; Semi-structured: The interviewer can ask other questions and prompts in addition to the specified questions; Unstructured: The interviewer has a list of topics, but no or few specified questions Interview guide A list of topics to be covered in an interview which is similar but less structured than a questionnaire and without multiple-response questions. Interview schedule A formal instrument that specifies the precise wording and ordering of all the questions to be asked of each respondent during a structured interview Interview protocol Data-collection instrument used in an interview Interviewee The person being asked questions Interviewer The person asking the questions; The assessor who conducts the interview, either in a ace-to-face setting or by telephone, and makes a record of the responses Intra-coder reliability Consistency within a single individual in coding qualitative data Intra-rater or Intra-observer reliability or agreement The degree to which the same rater or observer gives consistent estimates of the same phenomena; Achieved when the same coder arrives at the same results when the same body of material or level of phenomena is examined at different times or under different conditions Inventory A questionnaire or checklist, usually in the form of a self-report, that elicits information about an individual's personal opinions, interests, attitudes, preferences, personality characteristics, motivations, and typical reactions to situations and problems; A catalog or list for assessing the absence or presence of certain attitudes, interests, behaviors, or other items regarded as relevant to a given purpose Investigation A systematic examination, observation, or inquiry; May be a type of assessment task or activity, or a process conducted during or after the administration of an assessment as part of a quality check Investigator triangulation The use of multiple investigators in collecting and interpreting the data Investment center An enterprise that has a manager who is responsible for profit and investment performance and who controls most of the factors affecting revenues, costs, and investments Involvement Participating in or being engaged in an activity or relationship; In higher education: Student, faculty, administrator, alumni, or employer participation in the teaching and learning enterprise Ipsative scale

Assessment Glossary


86

Assessment or measurement by comparison with one’s self; Involves the individual using his/her values or behaviors as the yardstick by which the comparisons are made IQ or I. Q. tests The first of the standardized norm-referenced tests, developed during the nineteenth century; Traditional psychologists believe that neurological and genetic factors underlie "intelligence" and that scoring the performance of certain intellectual tasks can provide assessors with a measurement of general intelligence; There is a substantial body of research that suggests that IQ tests measure only certain analytical skills, missing many areas of human endeavor considered to be intelligent behavior; An IQ is considered by some to be fixed or static, but an increasing number of researchers are finding that intelligence is an ongoing process that continues to change throughout life Irregularity A variation form established rules, standards, principles, or procedures in a manner that can lead to unjustifiable actions or indefensible decisions Irrelevant information Information which is not coded or recorded because it is viewed as not relevant to the research question; In content analysis one must decide what to do with the information in the text that is not coded: One's options include either deleting or skipping over unwanted material, or viewing all information as relevant and important and using it to reexamine, reassess and perhaps even alter the one's coding scheme Item An individual question or exercise in an assessment or evaluative instrument; A single question, problem, or task used to assess a student; A question or problem on a test Item analysis A technique employed to analyze student responses to objective test items; Analyzing each item on a test to determine the proportions of students selecting each answer; Can be used to evaluate student strengths and weaknesses; May point to problems with the test's validity and to possible bias; Used both to improve the quality of items and to enhance the interpretation of results; May show the difficulty of the items and the extent to which each item properly discriminates between high achieving and low achieving students Item banks or pools A collection of test questions with known statistical properties from which a sample of items can be drawn to create a measure Item bias An item is biased when it systematically measures differently for different ethnic, cultural, regional, or gender groups Item characteristic curve (ICC) The graphical representation of item characteristics determined through the application of item response theory Item response theory (IRT) Test theory approach based on the observation that the probability of an examinee correctly responding to an item is determined by an interaction of examinee ability and item characteristics; A method for scaling individual items for difficulty in such a way that an item has a known probability of being correctly completed by an adult of a given ability level Item stem The set of words forming a question or statement Iterative

Assessment Glossary


87

Term used in research to refer to the repetition of a cycle of processes with an eye toward moving ever more closely toward desired results

Assessment Glossary


88

J Jackknife A resampling technique in which researchers rerun analyses several times dropping one participant from the analysis each time Job analysis A technique for studying a teaching job in terms of the knowledge, skills, and abilities (KSAs) needed to perform the job in a minimally acceptable manner as well as the functions and tasks that are performed as part of the job; May involve observation of teachers while they are doing their jobs as well as interviews with these teachers, and possibly their supervisors, and review of such documents as job descriptions, school procedures, and teacher products Job description A summary of the qualifications, duties, responsibilities, physical and mental demands, and working conditions associated with a specific job Johari window A leadership disclosure and feedback model used in performance measurement featuring the four quadrants or windows of knowing (Quadrant I represents the area of free activity or public area and refers to behavior and motivation known to self and known to others; Quadrant II represents the blind area where others can see things in ourselves of which we are unaware; Quadrant III represents the avoided or hidden areas, things we know but do not reveal to others; Quadrant IV represents the areas of unknown activity, in which neither the individual nor others are aware of certain behaviors or motives) Joint Committee on Standards for Educational Evaluation Coalition of professional associations founded in 1975 and accredited by the American National Standards Institute (ANSI) in 1989 that have published evaluation standards in the areas of accuracy, feasibility, propriety, and utility for use in student, program, and personnel evaluations (see http://www.wmich.edu/evalctr/jc/) Joint employment The relationship between a professional employer organization or employee leasing firm and an employer, based on a contractual sharing of liability and responsibility for employees Journals or journaling Students' personal records and reactions to various aspects of learning and developing ideas; A reflective process often found to consolidate and enhance learning; A way to assess student learning by having students record their new learning, questions to pursue, and observations about themselves as learners. Judgment An appraisal, decision, or opinion about the performance level of a learner/teacher with respect to the knowledge, skill, ability, behavior, or attribute being assessed Jurisdiction The limits or territory within which authority may be exercised; In bureaucratic politics, the area of programmatic responsibility assigned to an agency by the legislature or chief executive; A term used to describe the territory within the boundaries of a government entity (e.g. a local jurisdiction) Jury trial for projects Project evaluation process patterned after jury trial procedures for clarifying issues, introducing and assessing evidence, and reaching conclusions; Sometimes known as the Adversary Model of Evaluation

Assessment Glossary


89

Just in time An inventory control and management strategy in which raw materials are delivered from the supplier just prior to their use in the manufacturing process

Assessment Glossary


90

K k The size of the sampling interval used in systematic sampling Kendall coefficient of agreement (u) A test to detect an underlying logical pattern in a series of repeated paired comparisons Kendall coefficient of concordance (W) A test for the correlation of more than two variables used for ordinal data Key informant interviews A method of collecting information about a community or group of people by finding and interviewing a limited number of individuals determined or expected to be well informed and unbiased Key performance indicators (KPI) Quantifiable, specific measures of an organization’s performance in certain areas of its business; The purpose of KPIs is to provide the company with quantifiable measurements of items it has determined to be important to the organizational or business long-term goals and critical success factors; Used to understand and improve organizational performance and overall success; Also referred to as key success indicators Knowledge The sum of the information and experience the learner/teacher has acquired or learned and is able to recall or use; Cognitive objective that involves recalling or remembering information without necessarily understanding it and includes behaviors such as describing, listing, identifying, and labeling Knowledge assets The parts of an organization’s intangible assets relating specifically to knowledge, expertise, information, ideas, best practices, intellectual property, and other capabilities Knowledge integration Broadly defined as the assimilation, extraction, transformation, and loading of information from disparate systems into a single, more unified, consistent, and accurate data store used for evaluating, manipulating, and reporting information Knowledge management (KM) The process of creating, acquiring, sharing and managing knowledge to augment individual and organizational performance Knowledge mapping A process used to create a summation of the knowledge an organization will need in order to support its overall goals, objectives, strategies, and missions Known groups evidence Evidence that groups that are known to differ on the construct do differ on the test in the hypothesized direction KSAs Acronym for knowledge, skills, and abilities Kuder-Richardson formula or reliability coefficient (KR20, KR21)

Assessment Glossary


91

A statistical formula used to compute an estimate of the reliability of a homogeneous test; One of many formulas for determining the internal consistency of a dichotomously scored measure; A measure of inter-item consistency

Assessment Glossary


92

L Labor productivity The correlation between a given output and the percentage of labor time used to produce the output Last-in, first-out (LIFO) An asset-management and valuation system that assumes the assets acquired last will be used, sold or disposed of first; See also first-in, first out (FIFO) Late mover advantage The competitive advantage held by firms that are late in entering a market; Late movers are often able to imitate the technological advances of other firms or reduce risks by waiting until a new market is established Latent semantic analysis (LSA) A kind of machine learning that simulates the human brain's understanding of the meaning of words and paragraphs; Currently employed in the rating of short writing samples; Used in computerized grading of student writing; Proposed basis for a new generation of smart search engines Latent variable An indirectly observable variable or factor Lead time The total time required to produce a product or service Leadership The process by which an individual influences a group and directs it toward a specific goal or organizational mission Leading question A question that suggests a certain answer Lean manufacturing A management principle that focuses on the reduction of waste to improve overall customer value Learning The process by which a relatively lasting change in potential behavior occurs as a result of practice or experience Learning communities Concurrent student enrollment in two or more disciplines connected by a common theme; Sometimes targeted to a particular student group such as entering freshmen Learning curve A graph that depicts the relationship between the rate at which knowledge or a skill is learned and the time spent acquiring it Learning gain A positive change in learning outcomes measured following instruction or educational experiences; Difference between pretest and posttest; Longitudinal change in learning outcome measures Learning organization or institution

Assessment Glossary


93

An organization where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together; An organization that has developed a continuous ability to learn, adapt and change Learning outcomes Specific knowledge or skills that students actually develop though their college experience; The products of instruction or exposure to new knowledge or skills; Outcome behaviors: Observable behaviors or actions on the part of students that demonstrate that the intended learning objective has occurred; Operational statements describing specific student behaviors that evidence the acquisition of desired knowledge, skills, abilities, capacities, attitudes, or dispositions; Behavioral criteria for determining whether students are achieving the educational objectives of a program, and, ultimately, whether overall program goals are being successfully met; Often treated as synonymous with objectives, although objectives are usually more general statements of what students are expected to achieve in an academic program which are less measurable than outcomes Learning standards Learning standards define in a general sense the skills and abilities to be mastered by students in each strand at clearly articulated levels of proficiency Learning styles Different preferred approaches to learning (e.g. visual, auditory, kinesthetic, etc.) Legislation A proposed or enacted law or group of laws Leniency A type of rater effect in which an assessor or an evaluator tends to rate a learner/teacher too high or to judge the performance level as better than it actually is Lesson The content that is to be taught or the activity that is to be done during a specific period of instructional time Letter grade A summary evaluation of a student's proficiency or competency expressed on an alphanumeric or numeric scale (e.g. the common grading scale of A = 4, B = 3, C = 2, D = 1, and F = 0) Level Category into which a factor has been divided Level of analysis Chosen by determining which word, set of words, or phrases will constitute a concept in a content analysis; Generally 100-500 concepts is considered sufficient when coding for a specific topic Level of confidence The probability that a confidence interval to be constructed from a random sample will include the population parameter Level of generalization In content analysis a researcher must decide whether concepts are to be coded exactly as they appear, or if they can be recorded in some altered or collapsed form Level of significance The probability that observed or greater differences occurred by chance

Assessment Glossary


94

Level of implication In content analysis one must determine whether to code simply for explicit appearances of concepts, or for implied concepts, as well Level of measurement The relationship among the values that are assigned to the attributes for a variable; Knowing the level of measurement helps you decide how to interpret the data (e.g. when a measure is nominal then you know that the numerical values are just short codes for the longer names); Knowing the level of measurement helps you decide what statistical analysis is appropriate (e.g. if a measure is nominal, then you know that you would never average the data values or do a t-test on the data); A hierarchy is implied in the levels of measurement: At the lower levels assumptions tend to be less restrictive and data analyses tend to be less sensitive, at each level up the hierarchy, the current level includes all of the qualities of the one below it plus something new; There are typically four levels of measurement: Nominal: Numerical values just "name" the attribute uniquely with no implied ordering of the cases; Ordinal: Attributes can be rank-ordered but distances between attributes do not have any meaning nor do averages; Interval: The distance between attributes has meaning, the interval between values is interpretable, and averages have meaning but measurement ratios do not have meaning (e.g. 80 degrees is not twice as hot as 40 degrees); Ratio: Has an absolute zero that is meaningful and averages and fractions (or ratios) are meaningful; Generally it is more desirable to have a higher level of measurement (e.g. interval or ratio) rather than a lower one (nominal or ordinal) Licensing agreement An agreement between two enterprises or individuals granting the legal right to use a patent or trademark Lifelong education Students committed to lifelong learning as a result of their college experience, will express an openness and desire for intellectual experiences, express confidence in their ability to independently acquire new knowledge, and project a life of learning activities; Intellectual curiosity or intrinsic motivation to learn Lifelong learning A broad concept where education that is flexible, diverse, and available at different times and places is pursued throughout life; Concept associated with the four pillars of education for the future: learning to know, learning to do, learning to live together (and with others), and learning to be; The need for incumbent workers to continuously develop new technical skills and competencies as industry and technology changes and/or as workers change jobs Lifelong learning skills A skill that can be used in virtually all aspects of life or can be used in many situations throughout a person's lifetime (e.g. the ability to make and carry out effective plans) Likert scale A method to prompt a respondent to express their opinion on a statement being presented; Likert scales are often 4 point scales (strongly agree, agree, disagree, strongly disagree) or 5 point scales (strongly agree, agree, neutral, disagree, strongly disagree), but sometimes can have as many as 10 potential choices; An item type used on objective measures allowing respondents to indicate their level of agreement with a statement by marking their response on a five point scale, usually ranging from strongly agree to strongly disagree; A summated rating scale Line-item budget Budget format for listing categories of expenditures along with amounts allocated to each Linear regression A linear equation for forecasting criterion scores from scores on one or more predictor variables; A procedure often used in selection programs or actuarial prediction and diagnosis

Assessment Glossary


95

Listening skills Attending to and interpreting verbal messages from other people Literature review An in-depth review of key works and information available on a topic; A researcher typically conducts a literature search and uses the resulting information to write a literature review; The section of a research report that provides the framework of the research investigation and summarizes the literature the researcher studied in order to develop the research study Literature search A comprehensive survey of publications on a specific topic that usually results in a list of references; The process of conducting a literature search familiarizes a researcher with the body of work related to the research interest; A researcher may use the list of references from a literature search as source material to write a literature review Litmus test A test that uses a single key indicator to prompt a decision Loaded question A question containing emotionally charged words Local assessment Means and methods that are developed by an institution's faculty based on their teaching approaches, students, and learning goals Local norms Norms that have been obtained from data collected in a limited locale, such as a school system, county, or state; May be used instead of, or along with, national norms to evaluate student performance Local responsiveness The global strategy that emphasizes responsiveness to local markets and needs Localization The strategy of applying locale-specific terminology and data to a specific product or application in order to meet the language, cultural, and other requirements of a specific market Locally developed exams Objective and/or subjective tests designed by the faculty of the program or course sequence being evaluated Location parameter A statistic from item response theory that pinpoints the ability level at which an item discriminates, or measures, best Long-term assets The value of a company’s non-liquid assets such as property, equipment, and other capital minus depreciation Long-term debt Loans and other financial repayment obligations with a maturity date in excess of one year Long-term liabilities Debt and other repayments recorded on the balance sheets which are due in more than one year Long-term orientation

Assessment Glossary


96

The cultural dimension that values building for the future and making investments that will be successful over time Longitudinal Takes place over time Longitudinal data Data collected on the same individuals over time for use in a longitudinal study; Data collected from the same population at different points in time Longitudinal research Data are collected at multiple time points and comparisons are made across time Longitudinal study A study that investigates development, learning, or other types of change in individuals or other variables over time; An intervention study where treatment effects are reassessed at intervals over many years, not just at the end of the initial intervention period; Considered the preferred method of evaluating educational initiatives where the final results need to work their way in real time through the normal developmental life-cycle of the subjects; Examples include cohort, cross-sectional, ethogenic, and trend studies Low context cultures Cultures that primarily value clear and unambiguous messages and emphasize time management, deadlines and punctuality Low inference The types of assessment tasks, judgments, decisions, and conclusions that require a low degree of subjectivity on the part of the judge or evaluator Low inference descriptors Description that is phrased very similarly to the participants’ accounts and the researchers’ field notes Lower limit The smallest number on a confidence interval

Assessment Glossary


97

M Main effect The effect of one independent variable Major assessment Degree of student learning due to academic baccalaureate programs; See also Program assessment Management by objectives (MBO) Management technique, outlined by Peter Drucker in the 1950s, to increase organizational performance by aligning goals and subordinate objectives throughout the organization through employee participation in setting performance goals, courses of action, and time lines followed by performance measurement and evaluations; Requires SMART goals, management buy-in, a systems perspective, alignment with the corporate or organizational culture, and ethical safeguards and oversight; A management technique designed to facilitate goal and priority-setting, development of plans, resource allocation, monitoring progress towards goals, evaluating results, and generating and implementing improvements in performance Management consultant An individual who works independently to assist and advise clients with managerial responsibilities regarding various organizational issues Manipulation An intervention studied by an experimenter MANOVA (Multivariate Analysis of Variance) An extension of ANOVA methods to cover cases where there is more than one dependent variable and where the dependent variables cannot simply be combined; Technique used for assessing group differences across multiple metric dependent variables simultaneously, based on a set of categorical (non-metric) variables acting as independent variables; Multivariate significance test analogues of univariate ANOVA experimental designs; A test of the significance of group differences in some m-dimensional space where each dimension is defined by linear combinations of the original set of dependent variables; Analysis of variance where multiple dependent variables are bundled together into a weighted linear combination or composite variable (e.g. canonical variates, roots, Eigen values, vectors, or discriminant functions) Map A chart that summarizes the major elements of a system and shows the relationships between the parts of a system Margin of error One half of the width of a confidence interval Market life cycle The period of time during which a considerable portion of the buying public remains interested in purchasing a given product or service Market penetration The degree to which a product is recognized and purchased by customers in a specific market Market segmentation The process of dividing a market with a purpose of gaining a major portion of sales in a subgroup within a category instead of a more limited share of purchases by all category users

Assessment Glossary


98

Market share The percentage of the total sales (from all sources) of a given type of product or service attributable to a specific company Market value The most probable price (in terms of money) that buyers are willing to pay for particular goods or services Marginal mean The mean of scores in the cells of a column or a row Master list A list of all the codes used in a research study Mastery testing An assessment process designed to determine whether a teacher or student has acquired a predetermined level of content knowledge or skills; A test designed to indicate that the participant has or has not mastered some domain or knowledge or skill, generally indicated by a passing score or cut score Matched t-test A statistical test used to compare two sets of scores for the same subject; A matched pairs t-test can be used to determine if the scores of the same participants in a study differ under different conditions; For example: A matched t-test could be used to determine if people write better essays after taking a writing class than they did before taking the writing class Matching Equating the comparison groups on one or more variables that are correlated with the dependent variable; Process of corresponding variables in experimental groups equally feature for feature; An experimental procedure in which the subjects are so divided, by means other than lottery, that the groups are regarded for the purposes at hand to be of equal merit or ability Matching variable The variable the researcher uses in the control technique called matching in order to eliminate it as an alternative explanation Materials-based assessment Evaluation of learners on the basis of tests following the completion of a particular set of curriculum materials; A commercial text and its accompanying workbook is an example of this type of assessment Materials evaluation Evaluations that assess the merit or worth of content-related physical items, including books, curricular guides, films, tapes, and other tangible instructional products Matrix organization An organizational structure where employees report to more than one manager or supervisor Maturation Any physical or mental change that occurs over time in a participant and affects the participant’s performance on the dependent variable Maturity To evolve to full growth and development, stability, integration, allocentrism, autonomy, and symbolization

Assessment Glossary


99

Maximum likelihood estimation (MLE) A method of estimating population characteristics from a sample by choosing the values of the parameters that will maximize the probability of getting the particular sample actually obtained from the population Maximum variation sampling Purposively selecting a wide range of cases MBO See Management by objectives MBO-based evaluation A teacher evaluation approach based on a set of pre-specified objectives prepared for or in collaboration with the teacher Mean Arithmetic mean; The arithmetic average of a set of numbers; Standard measure of central tendency used for interval or ratio data; Also called "the first moment" or "center" of a distribution of numeric scores; For a chosen variable, sum the scores for each case and divide the sum by the number of cases and the result is the arithmetic mean; A measure of central tendency calculated by dividing the sum of all the values by the number of the values; May be overly affected by extremely low or high scores Mean deviation A measure of variation that indicates the average deviation of scores in a distribution from the mean; Determined by averaging the absolute values of the deviations. Measurability Refers to how the outcome is operationally defined and measured, including the methodological soundness of the chosen measures Measure A standard procedure for quantifying a sample of behavior from a larger domain; Often used interchangeably with test and instrument; An instrument or device that provides data on the quantity or quality of that aspect of teaching performance being evaluated; To classify or estimate, in relation to a scale, rubric, or standard, the degree of quality or quantity of that aspect of learning/teaching being evaluated; The process of collecting data using appropriate techniques Measure of variability A numerical index that provides information about how spread out or how much variation is present Measurement Assigning numbers to observations according to rules; The process of assigning numbers or categories to performance according to specified rules; The numerical estimation of the ratio of a magnitude of an attribute to a unit of the same attribute; May be quantitative or qualitative; Honest collection, storing, retrieving and interpreting of information to help an institution meets its mission; May be a frequency count of kinds of responses, a rating indicating the level of attainment of a desired quality, or the percentage of correct answers on a multiple choice test; In criterion-referenced assessment, may be specified as degrees of attainment based on what is possible and stated in terms of percentages of attainment along the continuum or section of a continuum (such as what can be expected for a particular grade level), where the continuum represents ultimate mastery of the criterion; Quantitative description of student learning and qualitative description of student attitude; Design of strategies, techniques and instruments for collecting feedback data that evidence the extent to which students demonstrate the desired behaviors; A quality or quantity arrived at by observation of an operationalized measure of a variable in known circumstances Measurement-based evaluation

Assessment Glossary


100

A teacher evaluation approach that is based on the use of methods which are not dependent on the expertise of the evaluator Measurement error Difference between the observed value of a measurement and the “true” value Measurement error variance That portion of the observed score variance attributable to one or more sources of measurement error; The square of the standard error of measurement Measures of central tendency The single numerical value considered most typical of the values of a quantitative variable (e.g. mode, median, and mean); Sometimes called a measure of "central location"; A single category or "most typical" score from the distribution of your variable Measures of relative standing Numerical values which provide information about where a score falls in relation to the other scores in the distribution of data Median The middle score in a distribution or set of ranked scores; The middle value in a series of values arranged in rank order; The point on a scale that divides a group into two equal subgroups; Not affected by extremely low or high scores; Measure of central tendency used with ordinal, interval, or ratio data; When a set of cases have been rank-ordered (from highest to lowest or lowest to highest) the median is the middle score; The median is also the 50th percentile; Cases or categories MUST be rank ordered or the median is meaningless; With true ordinal data the median category should be a verbal category; Do NOT use a numerical category from your output if the median category is a word or words, rather than a true interval-level number; It is a very common mistake to assign a number for the median to ordinal data; Use the verbal label for the median category in ordinal data; For example: "Agree" is a word, not a number Mental models A group or network of interrelated concepts that reflect conscious or subconscious perceptions of reality; Internal mental networks of meaning constructed as people draw inferences and gather information about the world Mentor teacher An experienced, often specially trained, teacher who works with new teachers, interns, or regular teachers in a professional improvement program; Mentors serve as resources, coaches, advisors, and confidants to other teachers and may be involved in formative evaluation activities as well as in the development and implementation of assistance plans Mentoring The provision of support by experienced teachers to promote the development of new or less experienced teachers Merit pay Salary increments allocated to a teacher based on an evaluation that demonstrates the teacher's superior level of performance; Increases in salary and wages that are tied to actual quality of work performed Meritorious performance Performance level significantly exceeding the minimally acceptable standard which may be worthy of professional recognition, career advancement, reward, or merit pay Meta analysis

Assessment Glossary


101

A systematic way of compiling results across studies in order to clarify the findings in an area of research; A quantitative technique used to integrate and describe the results of a large number of studies Meta cognition or metacognition The knowledge of one's own thinking processes and strategies, and the ability to consciously reflect and act on the knowledge of cognition to modify those processes and strategies; Refers to an individual's ability to think about his/her own thinking and to monitor his/her own learning; Metacognition is integral to a learner's ability to actively partner in his or her own learning and facilitates transfer of learning to other contexts Meta evaluation An evaluation of an evaluation Method of assessment Tests and procedures used to measure student performance in meeting the standards for a learning outcome; The techniques or instruments used to measure learner/teacher attributes and behaviors; For example: Rating scales, observation checklists, structured interviews, and portfolios Method of data collection The specific means used to document learner/teacher performance; Includes the data forms and procedures necessary to define the specifics of the valuation model or system; Five common methods are test scores, observation, reflection, ratings, and portfolios; Technique for physically obtaining data to be analyzed in a research study Method of evaluation or assessment The approach used to conduct the evaluation (e.g. the use of formal classroom observations followed by an interview with the supervisor and an oral examination by a team of peers); Techniques or instruments used in an assessment or evaluation Method of working hypotheses Attempting to identify all rival explanations Methods triangulation The use of multiple research methods Methodology How you collected the data; A set of analytical methods, procedures and techniques used to collect and analyze information appropriate for evaluation of the particular program, project or activity; Methodology is also a section of a research report in which the researcher outlines the approach used in the research, including the method of recruiting participants, the types of questions used, etc. Metrics The written, active, and operational components of an objective which are well defined and measurable in their accomplishment Mini-max budgeting A budgeting tactic in which two separate budgets are developed based on maximum assumed outputs and minimum assumed outputs Minimally acceptable A performance level that meets the minimum standards, as defined by its criteria; Any lower level of performance is not acceptable in terms of the purpose of an evaluation Minimum competency

Assessment Glossary


102

A level of knowledge, skill, or ability (usually demonstrated on a measure) that has been determined to be the minimum required for successful use of that knowledge, skill, ability, or personal trait Minimum efficient scale The smallest output for which unit costs are minimized Mission or mission statement A general statement of purpose guiding the practices of an institution or program; A program mission statement should provide an overview of the department/program’s philosophy, goals, and objectives, and embody the program’s purpose and the faculty’s priorities for the program; An institutional mission statement is the internally defined statement that documents the vision of the institution and answers the question: What is the organization attempting to accomplish?; Institutional mission statement objectives are broadly defined statements of intention defined by the strategic planning process which are not limited to student learning outcomes; A statement illustrating what the company is, what the company does and where the company is headed; A brief, comprehensive statement of purpose of an agency or program; Declaration of an organization’s purpose or reason for being; The broadest possible statement of the organization's purposes, goals, values and functions; Describes the underlying design, aim, or thrust of an organization and contains the basic purpose or philosophy of the organization Mixed or mixed method research Use of both quantitative and qualitative data collection and data analysis techniques; Research in which the researcher uses the qualitative research paradigm for one phase of a research study and the quantitative research paradigm for a different phase of the study; A quantitative phase and a qualitative phase are included in the overall research study; Research in which quantitative and qualitative techniques are mixed in a single study; Research that involves the mixing of quantitative and qualitative methods or paradigm characteristics Mixed model research Research in which the researcher uses both qualitative and quantitative research within a stage or across two of the stages in the research process; Quantitative and qualitative approaches are mixed within or across the stages of the research process Mixed purposeful sampling The mixing of more than one sampling strategy Mixed scanning A model of decision making that combines the rational-comprehensive model's emphasis on fundamental choices and long-term consequences with the incrementalists' emphasis on changing only what needs to be changed in the immediate situation Mode The score or value that occurs most frequently in a distribution; The most frequently occurring value in a set of ranked data observations; A measure of central tendency usable for all types of data; The category that contains the largest frequency or the greatest number of scores; The mode is the score that occurs the most often in your variable of interest; If your variable is a nominal measure, the mode is the only measure of central tendency that you can use (but you still can do percents, rates, ratios and compare groups on a nominal variable) Model An example of a coherent method, approach, procedure, or strategy of learning/teaching or of assessment/evaluation, as defined by its key or unique assumptions, propositions, attributes, supportive theory, research, practical precedent, or foundation, and which implicitly defines accomplished or good learning/teaching Modeling

Assessment Glossary


103

The use exemplary teachers and mentors to demonstrate practices of good teaching to other teachers for the purpose of improvement or of repertoire expansion Moderator The researcher who leads a focus group and facilitates discussion Moderator variable In regression analysis, a variable that serves to explain, at least in part, the correlation of two other variables; A variable that changes the relationship between other variables; A variable involved in an interaction effect Modern organization theory A body of theory emphasizing empirical examination of organizational behavior, interdisciplinary research employing varied approaches, and attempts to arrive at generalizations applicable to many different kinds of organizations Modifications Recommended actions or changes for improving student learning, service delivery, etc. that respond to the respective measurement evaluation Modus operandi analysis Deducing the cause of effects based upon analysis of events, process, or properties associated with the effects; Analogous to procedures used in detective work Monitoring The checking on a process or a person to verify that progress is being made, required activities are occurring, assessment and evaluation procedures are being implemented, suggested teaching practices are being tried, prior information is still applicable, earlier decisions can still be justified, and/or standards are being met; Continuous research which regularly measures the change in one or more indicators; Assessing the inputs and activities of a project Moral development The transformation that occurs in a person's form or structure of thought with regard to what is viewed as right or necessary Mortality Type of bias resulting from subjects not completing the study period; May be literal mortality (as in patient deaths in medical research) or figurative (as in student attrition in educational research); See Attrition Motivation The entire constellation of factors, both intrinsic and extrinsic, that causes a student to behave in a particular way at a particular time; With regard to student outcomes the focus is upon testing behaviors and the development of life-long learning Multi-factor test An instrument that measures two or more constructs which are less than perfectly correlated Multi-group research design A research design that includes more than one group of participants Multi-modal methods A research approach that employs a variety of methods Multi-trait multi-method

Assessment Glossary


104

An approach to construct validation where the construct of interest is measured using more than one experimental method and other related constructs are measured using the same methods so that a priori assumptions about the relationships between the constructs can be investigated Multidimensional assessment Assessment that gathers information about a broad spectrum of abilities and skills Multiple-baseline design A single-case experimental design in which the treatment condition is successively administered to different participants, or to the same participant in several settings, after baseline behaviors have been recorded for different periods of time Multiple choice or multiple response tests A test in which students are presented with a question or an incomplete sentence or idea, and are expected to choose the correct or best answer/completion from a menu of alternatives; A question, problem, or statement (called a stem) appears on a test followed by two or more answer choices, where the incorrect choices (called distractors) usually reflect common errors, and the examinee's task is to choose from the alternatives provided the best answer to the question posed in the stem Multiple measures The array of different types of evidence that are collected or assessment instruments that are used to better assess a learner/teacher's knowledge, skills, and performance; Multiple measures of the same attribute provide a more comprehensive, reliable, and valid measure of that attribute than any one measure alone; Multiple sources of assessment data gathered from a variety of sources Multiple operationalism The use of several measures of a construct Multiple regression Regression based on one dependent variable and two or more independent variables Multiple time-series design An interrupted time-series design that includes a control group to rule out a history effect Multiple-treatment interference Occurs when participation in one treatment condition influences a person’s performance in another treatment condition Multivariate analysis May be one of several methods for examining multiple variables; Strict usage: Research designs with two or more independent variables and two or more dependent variable; Loose usage: Research designs with two or more independent and/or dependent variables; Examples include path analysis, factor analysis, multiple regression analysis, MANOVA, LISREL, canonical correlations, and discriminant analysis Multivariate methods A collection of techniques appropriate for the situation in which the random variation in several variables has to be studied simultaneously; The main multivariate methods are principal components analysis, factor analysis, discriminant analysis, and cluster analysis Municipal bond An often tax-exempt bond issued by federal, city, county, state or local government agencies to finance capital expenditures for public projects Mutually exclusive

Assessment Glossary


105

Property that sets, categories, or intervals do not overlap; Response categories that are separate and distinct; Two or more events, conditions, or variables which cannot occur at the same time

Assessment Glossary


106

N n The sample size N The population size Narrative inquiry A qualitative research approach based on a researcher's narrative account of the investigation; Not to be confused with a narrative examined by the researcher as data Naturalistic assessment Assessment evaluation rooted in the natural setting of the classroom: Involves observation of student performances and behavior in an informal context done as students go about their daily work Naturalistic generalization Generalizing on the basis of similarity Naturalistic inquiry Observational research of a group in its natural setting; Qualitative researchers studying things in their natural settings, attempting to make sense of, or interpret, phenomena in terms of the meanings people bring to them Naturalistic observation Observation done in “real world” settings; One of the three basic approaches to scientific research which is characterized by recording of behavior as it occurs in a more or less naturalistic setting where there are no attempts at intervention National Institute for Literacy (NIFL) An independent federal organization created by the National Literacy Act of 1991 to serve as a focal point for public and private activities that support the development of high-quality regional, state, and national literacy services National Reporting System (NRS) An outcome-based reporting system for the state-administered, federally funded adult education program required by Title II of the Workforce Investment Act; The goals of the NRS were to establish a national accountability system for education programs by identifying measures for national reporting and their definitions, establishing methods for data collection, developing software standards for reporting to the U.S. Department of Education, and developing training materials and activities on NRS requirements and procedures Need Something that is wanted or required to increase a person's quality of life, general wellbeing or happiness; The difference between a desirable and an actual state of affairs; Needs may be classified as felt, expressed, normative, comparative, individual, group, organizational, clinical, administrative, subjective, and/or objective Needs analysis A method of analyzing how an employee skill deficit can be addressed through current or future training and professional development programs, how to determine the types of training/development programs required, and how to prioritize training/development Needs assessment

Assessment Glossary


107

A systematic exploration of the way things are compared to the way they should be; Evaluation of the service delivery capacity of a community or organization in relation to the current or future demand or need for those services; A process for identifying and prioritizing gaps in results based on the cost to meet the need versus the cost to ignore the need; A systematic study of a problem or innovation, incorporating data and opinions from varied sources, in order to make effective decisions or recommendations about what should happen next; A structured process to determine the needs of a designated survey field (e.g. individuals, an agency, a system, etc.) Negative case sampling Selecting cases that disconfirm the researcher’s expectations and generalizations Negative correlation The situation when scores on two variables move in opposite directions Negative predictive value (NPV) Diagnostic test that measures how well an instrument detects true negatives when all its decision negatives are considered; A high NPV indicates that the false negative problem is under control Negatively skewed Skewed to the left Negotiation The process of bargaining among parties to reach a mutually agreeable solution Network diagram A diagram showing the direct links between variables or events over time Neural network A computer driven statistical model that can learn to recognize patterns in data Next-in, first-out (NIFO) An inventory valuation method that values inventory at projected levels of cost that reflect replacement values over the near term No significant difference A decision that an observed difference between two statistics occurred by chance Noise Random error; Random outcomes or measures Nominal data A form of categorical data that classifies items into categories or groups, placing no real significance on the order of the groups Nominal group technique (NGT) Form of group discussion or decision-making where participants first work independently, then present their ideas one at a time to the group which then consolidates and ranks the ideas or proposed actions; A consensus planning tool used to identify the strengths of an organization, department, or division, whereby participants are brought together to discuss important issues, problems, and solutions Nominal scale A scale of measurement that uses symbols (such as words or numbers) to label, classify, or identify people or objects

Assessment Glossary


108

Nominal variable A variable determined by categories which cannot be ordered (e.g. gender and color) Non-directional alternative hypothesis An alternative hypothesis that includes the “not equal” sign Non-discrimination Evidence that differences of race or ethnicity, gender, or disability do not bias results of assessment instruments or procedures Non experimental research Research in which the independent variable is not manipulated and there is no random assignment to groups Non-instructional assessment Determining the extent to which service outcomes in support of student learning are achieved Non-instructional Unit Business units of a school that impact students in ways not directly in the classroom Non-linear regression A non-linear equation for forecasting criterion scores from scores on one or more predictor variables; A procedure often used in selection programs or actuarial prediction and diagnosis Nonequivalent comparison-group design A design consisting of a treatment group and a nonequivalent untreated comparison group both of which are administered pretest and posttest measures Nonparametric statistics A family of statistics that are based on the nominal categorization rank ordering of data points; These statistics do not require the sample to meet any distributional assumptions; Statistics which can be used with nominal and ordinal data; Also sometimes usable with interval and ratio data when other assumptions cannot be met; Examples include the chi-square test and the Spearman rank difference correlation coefficient; Mathematically "less powerful" than their parametric equivalents (less likely to lead to significant results than parametric tests) Nonreactive measures Assessments done without the awareness of those being assessed Nonverbal communication The non-spoken aspects of communication that include such culturally sensitive factors as kinesics (body language, gestures, eye contact, facial expressions, and other bodily movements), proxemics (conversational distance and other uses of space), paralanguage (tone, pitch, accent, and other nonverbal aspects of speech), chronemics (timing, frequency, tempo, rhythm, and other uses of time), haptics (touching), design factors (physical signals such as clothing, accessories, and interior design), etc. Norm A single value or a distribution of values constituting the typical performance of a given group; A distribution of scores obtained from a norm group; The midpoint or median of scores or student performance in that group; Performance standards that are established by a reference group and that describe average or typical performance; Usually norms are determined by testing a representative group and then calculating the group's test performance Norm group

Assessment Glossary


109

The group used as a point of comparison for a test; A random group of students selected by a test developer to take a test to provide a range of scores and establish the percentiles of performance for use in establishing scoring standards Norm referenced Describing a students’ performance on an assessment by comparison to a normed group; An interpretation of scores on a measure that focuses on the rank ordering of students not their performance in relation to criteria Norm referenced assessment An assessment where student performance or performances are compared to a larger group; Usually the larger group or "norm group" is a national sample representing a wide and diverse cross-section of students; Students, schools, districts, and even states are compared or rank-ordered in relation to the norm group; The purpose of a norm-referenced assessment is usually to sort students and not to measure achievement towards some criterion of performance Norm-referenced measurement Measurement that provides data on how well individuals do in relation to others in the group Norm referenced test interpretation A score interpretation based on a comparison of a test taker's performance to the performance of other people in a specified reference population Norm referenced test or testing A score that compares a student's performance to that of people in a norm group; The student or group scores will not fall evenly on either side of the median established by the original test takers which serve as a performance standard; Norm-referenced tests highlight achievement differences between and among studies to produce a dependable rank order of students across a continuum of achievement from high achievers to low achievers; An objective test that is standardized on a group of individuals whose performance is evaluated in relation to the performance of others Normal curve equivalent Arithmetically manipulated assessment data so that scores range from 1-99; Normalized standard scores with a mean of 50 and a standard deviation of approximately 21; Used to compare different tests for the same student or group of students and between different students on the same test Normal distribution The curve representing the normal distribution of a rating or test score; A unimodal, symmetric, bell-shaped distribution that is the theoretical model of many variables; A normal frequency distribution representing the probability that a majority of randomly selected members of a population will fall within the middle of the distribution; Represented by the bell curve; Also called a Gaussian distribution Normal distribution curve A bell-shaped curve representing a theoretical distribution of measurements that is often approximated by a wide variety of actual data; Often used as a basis for scaling and statistical hypothesis testing and estimation in psychology and education because it approximates the frequency distributions of sets of measurements of human characteristics Normal form The criteria applied to relational databases to insure good design and efficient management of the database; Following normal form avoids problems such as having the same data stored in multiple locations Normalized standard score A derived test score in which a numerical transformation has been chosen so that the score distribution closely approximates a normal distribution, for some specific population

Assessment Glossary


110

Normative forecasting A method of projecting future needs in order to determine what developments will be required to meet those needs Norming A process by which instructors come to an agreement about how the rubric will be used and standards are applied to evaluate assessments; Norming is done to avoid inter-rater error (i.e. large differences from evaluator to evaluator in how assessments are scored) and help achieve inter-rater reliability Norming group The specific group for which the test publisher or researcher provides evidence for test validity and reliability Norms Statistics or tabular data that summarize the distribution of test performance for one or more specified groups; Statistics that describe the test performance of specified groups, such as pupils of various ages or grades in a standardization group for a test; Norms may be designed or assumed to be representative of some larger population, such as pupils in the country as a whole; A performance standard that is established by a reference group and that describes average or typical performance; Usually norms are determined by testing a representative group and then calculating the group's test performance; A set of scores that describes the performance of a specific population of students at a particular grade level on a selection or constructed response set of tasks; The written and unwritten rules that specify appropriate group behavior Novice A person who has not had the education or training necessary to be considered an expert Null hypothesis The hypothesis of no difference or no differential effects; A statement about a population parameter which is a deliberate statement of the contrary of what you really suspect to be the case; Useful when applying inferential testing because group difference statistics, by their mathematical nature, make the initial presumption that two samples are from the same population until proven otherwise, and report when the means of the two samples move sufficiently apart for that presumption to be dismissed Numeric response A response style where the participant enters a number to indicate their choice Numerical rating scale A rating scale with anchored endpoints

Assessment Glossary


111

O Object of the evaluation What one is evaluating; For example: A program, project, or instructional material Objective A characteristic of an assessment, observation, or conclusion that minimizes the impact of bias and subjectivity, and that yields results which can be empirically verified Objective mastery Test items are written measuring particular objectives and if enough items measuring the same objective are answered correctly, then objective mastery is concluded Objective measures Tests or measures that can be scored without the need for subjective judgments (e.g. multiple-choice format) Objective percent correct The percent of the items measuring a single objective that a student answers correctly Objective test A test for which the scoring procedure is completely specified enabling agreement among different scorers; A correct-answer test; Style of testing that measures the participants knowledge of objective facts, the correct answers, to which, are known in advance Objective scoring Different scorers or raters will independently arrive at the same score or ratings for a student's performance; Most often associated with assessment methods comprised of selection items Objectives Objectives describe the skills, tools and/or content that students master which enable them to fulfill, perform, and/or accomplish the expected learning outcomes of a course or program of study; A specification of what is to be accomplished, the time frame in which it is to be accomplished and by whom; The specific knowledge, skills, or attitudes that students are expected to achieve through their college experience; Sometimes used to mean the same as expected or intended student outcomes: Intended results in precise terms which specify what needs to be assessed and serve as a more accurate guide in selecting appropriate assessment tools; The pre-specified intended outcomes of a program, process, or policy; Specific and measurable targets for accomplishing goals Objectives for learning See objectives Objectives-referenced test A test whose scores are referenced to the attainment of the objectives the test was designed to measure, rather than to the performance on the test by some comparison group of people Observable That which can be seen and documented by another person (e.g. the tone of the teacher's voice can be observed and recorded, but the thinking of the teacher that determined the tone of voice cannot be observed) Observation

Assessment Glossary


112

One of several methods used to collect data about a learner/teacher's performance; Includes the recording of evidence and notes while watching the behavior; May utilize audio or video recordings; A data collection procedure or tool in qualitative research; An everyday skill which is methodologically systemized and applied in qualitative research; Unobtrusive watching of the behavioral patterns of people; A research technique in which no direct questions are asked, but people are watched and their behavior recorded; A method of data collection in which data are gathered through visual or auditory observations; Observations may be structured or unstructured Observer The person who collects evidence and notes about what he/she is observing, either in a classroom or another setting; The observer is an assessor, but may or may not be an evaluator Observer effect The degree to which the assessment results are affected by the presence of an observer; The degree to which the presence of an observer influences the outcome Obtained score The actual results for a learner/teacher on an assessment; Due to measurement error and other factors, the obtained score may not be the same as the learner/teacher's true score Occupational exploration An awareness of a wide variety of occupations and the skills, knowledge, and attitudes needed to obtain them Occupational skills attainment Proficiency to perform actual tasks and technical functions required by certain occupational fields at entry, intermediate, or advanced levels OCR Optical Character Recognition; A method whereby a computer can recognize text and other marks that have been scanned Ombudsman Permanent office that receives complaints and acts on behalf of stakeholders to secure information, request services, or pursue grievances OMR Optical Mark Reader; A device that scans paper forms (normally bubble sheets) and recognizes the marks made on the form On-demand assessment An assessment that takes place at a predetermined time and place, usually under uniform conditions for all students being assessed; An assessment process that takes place as a scheduled event outside the normal routine; An attempt to summarize what students have learned that is not embedded in classroom activity; For example: The SAT, district and state tests, intake assessments, exit exams, and most in-class unit tests and final exams One-group pretest-posttest design A research design in which a treatment condition is administered to one group of participants after pretesting, but before posttesting on the dependent variable One-group posttest-only design Administering a posttest to a single group of participants after they have been given an experimental treatment condition

Assessment Glossary


113

One-stage cluster sampling A set of clusters is randomly selected and all of the elements in the selected clusters are included in the sample One-tailed test A directional application of one of the two-group inferential statistics (i.e. the t-test, the Wilcoxon, or the Mann-Whitney); Statistical procedure that only has to deal with the separation of two distributions down one or other of the asymptotes (or "tails" of a bell curve), but not both One-way analysis of variance Statistical test used to compare two or more group means Ongoing Addressed regularly as part of the business of the college rather than in response to periodic external requirements Ongoing assessment Tracks student learning by performance on tasks that are part of the natural lesson plan process; Ongoing assessment provides accumulated data for summative analysis by producing samples of work clustered by the outcome demonstrated Open coding The first stage in grounded theory data analysis Open-ended Assessment questions that are designed to permit spontaneous and unguided responses; A question that allows participants to respond in their own words; A format of a test, assessment, or survey item that calls for the answer to be supplied by the respondent rather than selecting from a list of options; For example: Essay questions, short-answer questions, drawings, and fill-in-the-blank items Open inquiry approach Form of evaluation which aims to improve or change projects by asking open-ended questions (e.g. “what's working?”, “what's not working?”, “how could we improve things?”, and “what are the community's needs?”) and using methods such as focus groups and interviews Open question See open-ended Open systems theory A theory of organization that views organizations not as simple, "closed" bureaucratic structures separate from their surroundings, but as highly complex, facing considerable uncertainty in their operations, and constantly interacting with their environment; Assumes that organizational components will seek an "equilibrium" among the forces pressing on them and their own responses to those forces Operating budget A detailed projection of all projected income and expenses during a specified future period Operating cash flow The total of company net profit, depreciation, accounts payable, and accrual changes, less inventory and accounts receivable changes Operating costs or expenses The day-to-day expenses incurred in running a business, organization, or project; Typically includes the costs of such items as personnel, materials, overhead, depreciation, and interest

Assessment Glossary


114

Operating grants Grants for use in the development and operation of specific programs Operational definition A definition of a term or object achieved by stating the operations or procedures employed to distinguish it from others; A very precise statement about how observed behaviors or events will be interpreted as representing a designated construct; Definitions for terms and research variables specific to one program or project; A definition used within a program or project Operationalism Representing constructs by a specific set of steps or operations Operationalize Defining a term or object so that it can be measured; Generally states the operations or procedures used that distinguish it from others; The defining of a psychological or physical attribute by the way it is measured; The act of assigning a particular physical dimension as a measure of a particular research variable Operating budget Tactical resource requirements, plus tactical priorities, equals the operating budget Operations management Decision-making that focuses on the performance and efficiency of the production process Opinion poll A type of survey in which people's opinions are asked Opportunity costs The amount that a particular product or resource could have earned in its next best use Opportunity or opportunistic sampling The act of selecting a research sample according to who is available to take part in it, rather than according to more precisely derived criteria; Selecting cases when the opportunity occurs Opportunity to learn (OTL) To provide students with the teachers, materials, facilities, and instructional experiences that will enable them to achieve high standards; What takes place in classrooms that enables students to acquire the knowledge and skills that are expected; May include what is taught, how it is taught, by whom, and with what resources Options Alternatives available to students to select from in multiple-choice items Oral examination An assessment of student knowledge levels through a face-to-face dialogue between the student and examiner-usually faculty Oral histories Interviews with a person who has had direct or indirect experience with or knowledge of the chosen topic Order effect A sequencing effect that occurs from the order in which the treatment conditions are administered Ordinal data

Assessment Glossary


115

Data or numbers that can be placed in rank order; The main difference between ordinal data and nominal data is that ordinal data contain both an equal-to and a greater-than relationship, whereas nominal data contain only an equal-to relationship Ordinal interaction effect An interaction effect that occurs when the lines on a graph plotting the effect do not cross Ordinal scale A rank-order scale of measurement Ordinal variable A variable in which the order of data points can be determined but not the distance between data points (e.g. letter grades) Organization chart A graphic representation outlining how authority and responsibility are distributed within an organization Organizational change A theory of organization that focuses on those characteristics of an organization that promote or hinder change; Assumes that demands for change originate in the external environment, and that the organization should be in the best position to respond to them Organizational culture An organization’s attitude and values regarding itself, employees, customers, and the general public; Encompasses the manner in which things are done within the organization based on defined policies and practices; Basic patterns of attitudes, beliefs, and values that underlie an organization's operation Organizational design The process of establishing and arranging the elements of an organization’s structure Organizational humanism A set of organization theories stressing that work holds intrinsic interest for the worker, that workers seek satisfaction in their work, that they want to work rather than avoid it, and that they can be motivated through systems of positive incentives (e.g. participation in decision making) Organizational or organization development (OD) Organization improvement through action research; A system-wide application of behavioral science knowledge to the planned development and reinforcement of organizational strategies, structures, and processes for improving an organization's effectiveness; Body of knowledge and practice that enhances organizational performance and individual development which views the organization as a complex system of systems that exist within a larger system, each of which has its own attributes and degrees of alignment; Collaborating with organizational leaders and their groups to create systemic change and root-cause problem-solving on behalf of improving productivity and employee satisfaction through improving the human processes through which they get their work done; A long range effort to improve an organization's problem solving and renewal processes, particularly through more effective and collaborative management of the organizational culture, often with the assistance of a change agent or catalyst and the use of the theory and technology of the applied behavioral sciences (primarily industrial/organizational psychology, industrial sociology, communication, cultural anthropology, administrative theory, organizational behavior, economics, and political science); A planned approach to far-reaching organizational change designed to enable an organization to respond and adapt to changing market conditions and to set a new agenda; A long-term, systematic, and prescriptive approach to planned organizational change which applies the concepts of the social and behavioral sciences to the problem of changing an organization and treats the organization as a complex social and technical system that should have enough flexibility to change its design according to the nature of its tasks and external environment; A theory of organization that concentrates on increasing the ability of an organization to solve internal problems of

Assessment Glossary


116

organizational behavior as one of its routine functions and concerned primarily with the identification and analysis of such problems Organizational planning The process of transforming an organization’s goals, objectives, philosophy and mission into practices and policies Organizational structure The intentional arrangement of positions in an organization to accomplish the goals of the company; Types of structures include functional, geographic, product, hybrid, and matrix Organizational transformation Organization-wide changes (such as restructuring operations, introducing new technologies, processes, services or products, implementing new programs, re-engineering, etc.) Orientational research Research done for the purpose of advancing an ideological position; Sometimes called critical theory research Out-of-level testing Administering a test that is designed primarily for people of an age or grade level above or below that of the test taker Outcome evaluation Evaluations that look specifically at whether or not the program achieved its goals and had its intended effect; See also Impact evaluation Outcome measure Instruments used for gathering information on student learning and development; A performance measure that quantifies the extent to which results are achieved, and that measures effectiveness Outcome validity The ability to generalize across different but related dependent variables Outcome variables The results or products of teaching; For example: Student mastery of skills, completion of courses, teacher-developed instructional materials, student projects, and student performance on standardized tests Outcomes An operationally defined educational goal, usually a culminating activity, product, or performance that can be measured; The specific knowledge, skills, or developmental attributes that students actually develop through their college experience; The "end-products" of the entire instructional process; Specific, measurable behaviors reflecting desired student knowledge, skills, values, or levels of service provided by a course, discipline, program, or institution; The long-term end goals of an institution, program, course, project, or department; Outcomes reflect the actual results achieved, as well as the impact or benefit, of a program; Assessment results Outcomes assessment A strategy used to evaluate and measure the results of an instructional method or program Outlier A number that is very atypical of the other numbers in a distribution Output or outputs Anything an institution or system produces; A type of performance measure that focuses on the level of

Assessment Glossary


117

activity in a particular program; Direct products of a program's activities which are usually measured in terms of the volume of work accomplished; For example: The number of classes taught, counseling sessions conducted, educational materials distributed, and participants served; Outputs have little inherent value in themselves but may lead to outcomes or benefits Output measure A performance measure that measures efficiency by quantifying how many products are produced or services performed Overall performance level A combination of the cut-scores or proficiency levels of the various assessments used to determine whether students do not meet, meet, or exceed the standard set for a whole learning outcome; Different assessments may be given greater weight when determining an overall performance level. Overview Conceptual/introductory statement that gives essential definitions, provides a general rationale, and presents summarized procedures, common problems, and special difficulties that are applicable

Assessment Glossary


118

P p-value Standard mathematical shorthand for reporting a probability; A number between zero and one tells us that the event may occur, and tells us if the event is more likely (a probability nearer to one, with 1 being certainty) or less likely (a probability nearer to zero, with 0 being totally improbable) to occur; If we are tossing a fair coin the probability of tossing a head is 0.5, and the probability of tossing a tail is also 0.5; The probability of an event ranges between zero and one, where zero probability means that the event will never occur, and a probability of one means that the event is certain to occur; Sometimes probability is described as a percent, ranging from 0% to 100%, instead of a p-value; May also be a test item statistic that represents the percentage of students who answered that item correctly out of a particular population group: May be calculated for a national standardization sample or for a class or school level population where a test has been administered; Calculated by dividing the number of correct responses on an item by the total number of students tested; May be expressed as a decimal value or as a percentage (by multiplying the decimal value by 100); Panel data Refers to data that is collected on the same variables across different cohorts (e.g. collecting incoming freshman SAT scores each year) Panel study The same individuals or group of respondents are repeatedly studied or surveyed at successive points in time to detect changes in their behavior or opinions Paradigm See Research paradigm Paradigm shift A term used to define the process of change in patterns of thought or behavior; A change in how we think about something Parallel forms Two or more forms of a test constructed to be as comparable and interchangeable as possible in their content, difficulty, length, and administration procedures and in the scores and test properties (e.g. means, variance, and standard error of measurement); Two (or more) item sets, matched for difficulty, in a psychometric package used to avoid practice effects which might prevent using the same assessment twice on the same subjects Parameter A numerical characteristic of a population; A coefficient or value for the population that corresponds to a particular statistic from a sample and is often inferred from the sample; A value used to represent a certain population characteristic; For example: the population mean is a parameter that is often used to indicate the average value of a quantity; A quantity, property or fact that gives information about the whole population; A population parameter is usually unknown and has to be estimated Parametric statistics Used with interval and ratio data and usually with data that were obtained from groups randomly assigned, normally distributed, and with equal variability between groups; Preferred statistics to use because they are more "powerful" than nonparametric statistics; Examples include: t-tests, analysis of variance, and Pearson correlation coefficient Pareto chart A bar graph used to rank in order of importance information such as causes or reasons for specific problems so that measures for process improvement can be established

Assessment Glossary


119

Parochialism A personal view of the world based only on one’s own perspectives and values; This view does not recognize or respect other perspectives or ways of living and working Partial correlation Used to examine the relationship between two quantitative variables controlling for one or more quantitative extraneous variables Partial regression coefficient The regression coefficient obtained in multiple regression Participant A person that participates in a testing, assessment, or survey process by answering questions Participant-as-observer Researcher spends extended time with the group as an insider and tells members they are being studied Participant feedback Discussion of the researcher’s conclusions with the actual participants Participant observation Observation of a group in which the observer is immersed in the day-to-day lives of the people or through one-on-one interviews with members of the group Participatory learning and action (PLA) An umbrella term for a wide range of similar approaches and methodologies involving the full participation of people in the processes of learning about their needs and opportunities, and in the action required to address them Participatory research A method in which practitioners and learners consciously participate together as researchers in the research process; Participatory research is often action research with a goal of achieving change Particularism A cultural perspective that is contingency-oriented and uses different evaluative standards based on relationships and situations Passing score A score or level of performance that represents the difference between those learner/teachers whose level of performance is minimally acceptable and those whose performance is not acceptable; Also called Cut or critical score Passive consent A process whereby consent is given by not returning the consent form Path analysis An application of multiple regression analysis used to develop and test causal models using correlational data Path coefficient A quantitative index providing information about a direct effect Pattern

Assessment Glossary


120

A series of similar behaviors or common attributes over a period of time or across different settings or contexts Pattern matching Predicting a pattern of results and determining whether the actual results fit the predicted pattern Pedagogy Instructional methods, usually not related to or specific to adult learning; The art and science of teaching; Some pedagogical skills apply across teaching situations whereas others apply only to specific subject areas Peer assessment An assessment method in which students within a similar educational setting make and report judgments about other students' performances; Evaluation of learning by one’s peers Peer institution An institution with similar or aspirational characteristics whose comparative quantitative data is used to establish goals, to set priorities, to measure progress, and to influence institutional improvement Peer review Used as part of an accreditation process, part of an academic or student affairs program development, or for an internal review process; The evaluation of an educational institution by other educators; Performance evaluations done by co-workers; The evaluation of a teacher by other teachers; Usually done to provide feedback to the evaluee for purposes of professional development and improvement or to provide subject-matter and context-related expertise; Discussing one’s interpretations and conclusions with one’s peers or colleagues; An evaluation of a study by recognized authorities in a subject area prior to publication Percent Score The percent of items that are answered correctly Percentile The percent of people in the norming sample whose scores were below a given score; A ranking scale ranging from a low of 1 to a high of 99 with 50 as the median score; A percentile rank indicates the percentage of a reference or norm group obtaining scores equal to or less than the test-taker's score; A percentile score does not refer to the percentage of questions answered correctly, it indicates the test-taker's standing relative to the norm group standard Percentile rank The percentage of examinees in the norm group who scored at or below the raw score for which the percentile rank was calculated; A number indicating an individual's performance level or score in relation to its standing in the distribution of scores of a representative group of individuals; A percentile rank of 95 means that the individual did as well as or better than 95% of the group upon whom the percentile ranks are based; Percentile ranks divide a distribution into 100 equal parts; Percentile ranks cannot be arithmetically manipulated due to their varying interval nature Perception Use of previous knowledge to gather and interpret the stimuli registered by the senses; Combines aspects of both the outside world (the visual stimuli) and one’s inner world (prior knowledge) Performance accountability A means of judging policies and programs by measuring their outcomes or results against agreed upon standards; A performance accountability system provides the framework for measuring outcomes - not merely processes or workloads Performance appraisal or review

Assessment Glossary


121

The systematic process of determining the merit, value, and worth of a teacher's current performance and estimating his/her potential level of performance with further development; Specific evaluation with respect to an individual's progress in completing specified tasks Performance auditing Analysis and evaluation of the effective performance of agencies in carrying out their objectives Performance based assessment Direct, systematic observation and rating of student performance of an educational objective, often an ongoing observation over a period of time, and typically involving the creation of products; May be a continuing interaction between teacher and student, and should ideally be part of the learning process and a real-world performance with relevance to the student and learning community; Broad term encompassing many of the characteristics of both authentic assessment and alternative assessment; Assessment technique involving the gathering of data though systematic observations of a behavior or process and evaluating that data based on a clearly articulated set of performance criteria to serve as the basis for evaluative judgments; An observation of the process of creating an answer or product that demonstrates a student's knowledge and/or skills; A formal assessment method in which a student's skill in carrying out an activity and producing a product is observed and judged; The process of using student activities or products, as opposed to tests or surveys, to evaluate students’ knowledge, skills, and development; Directly observable, student-generated evidence of learning (e.g. creating a object or product, performing a task or behavior, and interpreting or critiquing the products or performances of others); An assessment activity that requires students to construct a response, create a product, or perform a demonstration; Since performance-based assessments generally do not yield a single correct answer or require a particular solution method, evaluations of student performances are based on judgments guided by criteria using a rubric, or analytic scoring guide to aid in objectivity; Examples include: Reflective journals (daily/weekly); capstone experiences; demonstrations of student work (e.g. acting in a theatrical production, playing an instrument, observing a student teaching a lesson); products of student work (e.g. Art students produce paintings/drawings, Journalism students write newspaper articles, Geography students create maps, Computer Science students generate computer programs, etc.); Important elements include clear goals or performance criteria clearly articulated and communicated to the learner, the establishment of a sound sampling that clearly envisions the scope of an achievement target and the type of learning that is involved, attention to extraneous interference, and establishment of a clear purpose for the data collected during the assessment before the assessment is undertaken, while keeping in mind the needs of the groups involved Performance budget or budgeting Budget format organized around programs or activities (rather than the objects it purchases), including various performance measurements that indicate the relationship between work actually done and its cost; A budgeting system predicated on the relationship between inputs and outputs and aligning goals with key management activities Performance criteria or standard Set standards by which student performance is evaluated which help assessors maintain objectivity and provide students with a target or goal; Established levels of achievement, quality, or proficiency; Explicit definitions of what students must do to demonstrate proficiency at a specific level on the content standards which set expectations about how well students should perform; The observable aspects of a performance or product that are observed and judged in a performance assessment; Standards which help assessors maintain objectivity and provide students with important information about expectations by giving them a target or goal to strive for Performance evaluation The process of determining the merit, value, and worth, based on assessment results, of some performance attribute(s) of the teacher being evaluated Performance goal Specific statement of what is to be accomplished by an evaluee (e.g. growth in knowledge, development of a skill, changes in practice), how the goal will be met (e.g. activities, resources), when the goal will be met, and how achievement of the goal can be assessed or determined

Assessment Glossary


122

Performance indicators A set of measures with which postsecondary institutions, systems, states, or sectors evaluate and report their performance and often used to measure efficiency or effectiveness; Quantitative measures of success used to determine progress toward achievement of strategic goals Performance measures Tools or information used to measure results and ensure accountability; A yardstick or standard used to measure progress toward achieving a strategic or tactical objective; A test-taking method in which the participants perform some real-life behavior that is observed by the researcher Performance standards The levels of achievement students must reach to receive particular grades or to be allowed to move to the next unit in a criterion-referenced assessment system; Explicit definitions of what students must do to demonstrate proficiency at a specific level on the content standards; A statement or description of a set of operational tasks exemplifying a level of performance associated with a more general content standard; The statement may be used to guide judgements about the location of a cut score on a score scale; The term often implies a desired level of performance Performance task Assessment activity, or set of activities, related to one or more learning outcomes, that elicits one or more response to a question or problem; A carefully planned activity that requires learners to address all the components of performance of a standard in a way that is meaningful and authentic; Performance tasks can be used for both instructional and assessment purposes Performance test Designed to evaluate general intelligence or aptitudes; Consists primarily of motor items or perceptual items because verbal abilities play a minimal role Periodic budgeting A budget developed for the purpose of covering a specific period of time (e.g. annual budget) Periodicity The presence of a cyclical pattern in the sampling frame Persistence Continual enrollment in pursuit of a credential or credit Personal growth and change Aspects of students’ relationship systems, self-systems, cultural, intellectual education, occupational, political, social, religious values and activities and gender role orientations that are influenced by college attendance and completion Personal documents Anything written, photographed, or recorded for private purposes Personality The relatively permanent patterns that characterize and can be used to classify individuals Personnel evaluation The systematic determination of the merit, value, and worth of the job-related performance of an employee Petty cash Refers to the small quantity of cash typically kept on hand by a company to cover incidental expenses

Assessment Glossary


123

Phenomenological studies The examination of human experiences through detailed descriptions of the people being studied; Procedure involves studying a small number of subjects through extensive and prolonged engagement to develop patterns/relationships of meaning Phenomenology Study of objects and events as they appear to the experiencing observer; The description of one or more individuals’ consciousness and experience of a phenomenon; A qualitative research approach concerned with understanding certain group behaviors from that group's point of view; A form of qualitative research in which the researcher attempts to understand how one or more individuals experience a phenomenon Physical assets A term used to describe a company’s or individual’s tangible assets such as property, equipment, furnishings, inventory and supplies Physical data Any material thing created, altered, or left behind by people which might provide information about a phenomenon of interest to a researcher Pilot or pilot test Small scale trial of an assessment instrument to test its validity and usability before the full scale assessment is run; Small scale trial of a treatment on a sample population to test its effectiveness prior to a full scale implementation on the parent population; Small scale trial of a plan or program to test the design and procedures prior to a full scale implementation; A brief, simplified preliminary trial study designed to learn whether a proposed project, program, or evaluation seems likely to yield valuable results; A large-scale administration of an assessment, usually with several classes of students if not all students in a program, to detect any flaws in the assessment before the assessment is considered "done" and is fully implemented Pilot study A “scaled down” version of a major effort conducted before a larger study to test feasibility Pilot testing A brief, simplified preliminary trial study designed to learn whether a proposed evaluation seems likely to yield valuable results; A test administered to a representative sample of participants solely for the purpose of determining the properties of the test; A preliminary try-out of a new or revised assessment or process; Considers such areas as comprehensiveness and clarity of directions, format of assessment materials, adequacy of resources or equipment to be used for the assessment, quality of assessor/evaluator training programs, and timing of assessment tasks Pipeline analysis The analysis of student flows into, through, and out of an academic program; Includes analysis of retention rates, course-taking patterns, and graduation rates; May highlight: sources of student matriculations, time durations, and unit achievements while students are enrolled, typical paths of student course enrollments, illustrative trends in student flows over time, and potential targets for improved student flows Pitfall A not easily recognized difficulty believed to be associated with a particular standard; Mistakes that would be made out of ignorance of the import and intent of a standard Placebo A dummy treatment; A treatment known by the practitioner to have no effect on the condition or variable being measured but which the subject is told or believes to be effective

Assessment Glossary


124

Placebo group Group in an intervention study which is given a dummy treatment, and (usually) kept unaware of that fact; See also Control group Placebo effect The measurable, observable, or felt improvement in health or behavior not attributable to an administered medication or invasive treatment; Improvement in a measure recorded for the control group which was given a treatment known to have no effect on the measure Placement exams Instrument to determine if students’ qualifications for entering a program or course are at an appropriate level to begin their studies Plan of assistance or assistance plan A strategy for professional development and growth designed to address a teacher's deficiencies in meeting designated performance standards, based on the results of an evaluation; The plan of assistance should indicate goals and objectives for improvement, an action plan for improvement, what staff and resources are available, the timeline for development activities, benchmarks for ensuring that professional growth is occurring, and measures for verifying achievement of the goals and objectives Planning The development of a design or scheme of arrangement with a definite purpose; Institutional planning includes the educational programs, the physical plant, budgets and finances, and is intended to accomplish the purposes of the institution. Planning horizon The farthest date that the strategic plan addresses; The target date by which the organization expects to accomplish its intended outcomes Planning-programming-budgeting system (PPBS) Effort to connect planning, systems analysis, and budgeting in a single exercise Pluralism A social and political concept stressing the appropriateness of group organization and diversity of groups and their activities as a means of protecting broad group interests in society; Assumes that groups are good and that bargaining and competition among them will benefit the public interest Point estimate The estimated value of a population parameter Point estimation The use of the value of a sample statistic as the estimate of the value of a population parameter Policy The principles, plan, or procedures established by an agency, institution, or government, generally with the intent of reaching a long-term goal; Statement of goals and intentions with respect to a particular problem or set of problems; A written statement that reflects the employer’s standards and objectives relating to various employee activities and employment related matters Policy analysis Process of researching or analyzing public problems to provide policy makers with specific information about the range of available policy options, and the advantages and disadvantages of different approaches

Assessment Glossary


125

Political economy approach Focusing on politics and economies as categories for analyzing organizational behavior Politics of the budgetary process The requirement that administrators act as advocates for their own programs during the appropriation process by soliciting outside support, protecting their budgetary base, and inching ahead with new programs; A budgetary system that deals with complex problems by relying upon incremental methods of decision making, information drawn from past experience rather than analysis, and satisfactory rather than optimal standards of quality Poll Usually the same as opinion poll, but sometimes loosely used to mean any type of informal survey Population All the members of a uniquely definable group of people or things; The target group under investigation; A population is any entire collection of people, animals, plants or things from which we may collect data; The entire group we are interested in, which we wish to describe or draw conclusions about; Populations are often too large or expensive to measure directly and “sample” populations must be used to infer the characteristics of the “parent” population; The entire set of objects you are studying; Each member of a population is called an individual but need not be a person Population validity The ability to generalize the study results to individuals who were not included in the study Portal or community portal A Web-based application that provides personalization, single sign-on, and content aggregation from different sources and hosts the presentation layer of information systems; A group or community access gateway to the World Wide Web, usually requiring a secure login, which brings together local information and offers services like email and discussion forums to extend community activity and interaction; Portals integrate various information sources and services, such as search engines and directories using customized content, layout, and navigation to suit group or organizational needs; Local needs and interests are central to community portals Portfolio A collection of a student's work specifically selected to tell a particular story about the student; An accumulation of evidence about individual proficiencies, especially in relation to learning standards; A purposeful, integrated collection of student work showing effort, progress, or achievement in one or more areas; A systematic and organized collection of a student's work that exhibits to others the direct evidence of a student's efforts, achievements, and progress over a period of time; A collection of student work to show not only learning outcomes but also the progress or process of learning; The collection should involve the student in the selection of its contents; May include information about the performance criteria, the rubric, or criteria for judging merit; May include evidence of student self-reflection or evaluation; May include representative work, providing a documentation of the learner's performance and a basis for evaluation of the student's progress: May include a variety of demonstrations of learning gathered in the form of a physical collection of materials (e.g. videos, CD-ROMs, reflective journals, etc.); Portfolios may be used not only as a course-level assessment but also at the program-level to show learning progress throughout a whole program; May be a collection of student work over time showing change or a collection of student work showing breadth of learning outcomes; Program portfolios include collections of multiple student work samples usually compiled over time and rated using rubrics; The design of a portfolio is dependent upon how the scoring results are going to be used; Portfolios can be designed to assess student progress, effort, and/or achievement, demonstrate the student’s learning process and conscious involvement in the process, and encourage students to reflect on their learning Portfolio assessment Method of assessment that relies on a collection of student- and/or teacher-selected samples of student work or performance in order to evaluate individual student achievement; A portfolio becomes a portfolio assessment when the assessment purpose is defined, criteria are made clear for determining what is contained in the

Assessment Glossary


126

portfolio, and criteria for assessing either the collection or individual pieces of work are identified and used to make judgments about performance; Commonly assessors work together to establish a consensus of standards or to ensure greater reliability in evaluation of portfolios Position classification Analyzing and organizing jobs on the basis of duties, responsibilities, and the knowledge and skills required to perform them Positioning statement A statement defining a company’s target market, brand reference, need fulfillment, and needed support to communicate what the company stands for Post-test or posttest A test to determine performance after the administration of a program, project, or instructional material; The measurement of a dependent variable, which occurs after an intervention, usually for the purpose of comparing to a pretest measure on the same dependent variable Postsecondary education Program at an accredited degree-granting institution that leads to improved skills, a certificate of competency, or an academic degree; College or university education Positive correlation The situation when scores on two variables move in the same direction Positive predictive value (PPV) A measure of how good a test is at detecting true positives when all its decision positives are considered; A high PPV indicates that the false positive problem is under control Positively skewed Skewed to the right Positivist research paradigm Approach which states that only observable and measurable data should be taken into account in research; Positivism is an approach originally developed in the natural sciences and transferred to social studies; Positivist social science research uses random samples, operationally defined variables, and statistical analysis; Positivists attempt to use only data provided by direct observation to reach “positive” facts Post hoc fallacy Making the argument that because A preceded B, A must have caused B Post hoc test A follow-up test to the analysis of variance Posttest-only control-group design Administering a posttest to two randomly assigned groups of participants after one group has been administered the experimental treatment condition Posttest-only with nonequivalent groups design Comparing posttest performance of a group of participants who have been given an experimental treatment condition with a group that has not been given the experimental treatment condition Power The ability of an experimental design or inferential statistic to detect an effect of a variable when one is present; The likelihood of rejecting the null hypothesis when it is false

Assessment Glossary


127

Power distance A cultural value dimension that focuses on the degree to which power differences are accepted in a culture Power test A test that samples the range of an examinee's capacity in particular skills or abilities and places minimal emphasis on time limits; A test intended to measure level of performance without regarding speed of response; Test in which there is either no time limit or a very generous one; Items are usually arranged in order of increasing difficulty; A pure power test is sometimes defined as one in which every examinee has sufficient time to complete the test Practice The manner in which evaluations or assessments are actually conducted, whether or not the practice is in accordance with the policy and/or follows the procedures; What practitioners (e.g. researchers, teachers, office holders, etc.) actually do Practice effect Type of confounding possible when the measure in question is itself a learnable mental or physical skill Practical significance A conclusion made when a relationship is strong enough to be of practical importance Practitioner An individual who practices a learned profession Pragmatism The philosophical position that what works is what is important Pragmatist philosophy A philosophy that says to use or do what works Pre-test or Pretest A test to determine performance prior to the administration of a program, project, or instructional material; The measurement of a dependent variable prior to an intervention, usually for the purpose of comparing to a posttest measurement of the same dependent variable Pre-test/post-test or pretest-posttest An assessment technique in which students are given an assessment prior to a treatment or course (the pre-test) to provide a baseline measurement and are given the same or similar assessment at the end of the treatment or course (the post-test) to provide an impact measurement; Usually a valid means to show learning in a course because with prior knowledge established through the pre-test it is possible to show learning took place in the course itself by comparing the pre- and post-test results Precision In survey research, the tightness of the confidence limits Precision of measurement A general term that refers to the reliability of a measure, or its sensitivity to measurement error Predefined versus interactive concept In content analysis one must determine whether to code only from a pre-defined set of concepts and categories, or if one will develop some or all of these during the coding process

Assessment Glossary


128

Prediction Attempting to predict or forecast a phenomenon Predictive research Research focused on predicting the future status of one or more dependent variables based on one or more independent variables Predictive validity Ability of a measure to predict some future behavior; The ability of a score on one test to forecast a student's probable performance on another test of similar skills; The relationship of a measure to performance in a future context such as a new work setting or to the results obtained on a future measure assessing a similar or a different (but presumably related) attribute; A term used to refer to a type of criterion-related validity that applies when one wishes to infer from a test score an individual's most probable standing on some other variable called a criterion; Called criterion-related evidence to emphasize that it refers to one type of evidence within a unitary conception of validity; Refers to evidence based on relations to other variables that include test-criterion relationships; Predictive evidence indicating how accurately test data can predict criterion scores that are obtained at a later time; Used in the test validation process to measure the relationship between test scores and actual job performance Predictive validity evidence Validity evidence based on the relationship between test scores collected at one point in time and criterion scores occurring at a later time Predictor domain The construct domain of a construct used as a predictor Predictor variable Variable believed to predict the magnitude of a second variable; See Independent variable Predictors A set of variables believed to predict the magnitude of a second variable, using a criterion Premium pricing High pricing by a company because of the perceived superiority of a product or service or the wish to intimate superior quality Preponderance Emphasis or weight given to data and information on a teacher attribute in an evaluation, after considering the quantity and frequency of the behavior, the importance of the attribute to the job of teaching, and the potential impact of the behavior or characteristic on students Prerequisite knowledge The prior knowledge that is necessary in order to learn how to solve problems or to acquire new knowledge and skills Presence or absence technique Manipulating the independent variable by presenting one group with the treatment condition and withholding it from the other group Presentism The assumption that the present-day connotations of terms also existed in the past Pretest

Assessment Glossary


129

An administration of test items to a representative sample of participants solely for the purpose of determining the characteristics of the item Pretest-posttest control group design A research design that administers a posttest to two randomly assigned groups of participants after both have been pretested and one of the groups has been administered the experimental treatment condition Price escalation clause A contract provision calling for an increase in price should an increase in certain types of costs occur Primary or direct data Information related to a single student’s learning; Original data collected as a part of a research study Primary source A source in which the creator was a direct witness to, or in some other way directly involved in or related to, the event being studied Primary standards Those standards that apply to the evaluation process and assessment methods rather than to the learner/teachers being evaluated or to their performance levels, and which should be met or addressed before the assessments are administered and the evaluation process is implemented Primary trait analysis Assignment-specific factors or traits considered in scoring an assignment are stated in a hierarchical scale of three to five incremental levels of achievement quality, and for each level on the scale there is a specific statement that describes expected behavior (criterion) at that level; Used to help teachers develop their criteria for grading Primary trait method A type of rubric scoring constructed to assess a specific trait, skill, behavior, or format, or the evaluation of the primary impact of a learning process on a designated audience Primary trait scoring The assignment of scores to one or more designated attributes of each task or performance measure; Scoring procedure in which products or performances are evaluated by limiting attention to a single criterion; These criteria are based upon the trait determined to be essential for a successful performance on a given task and scorers measure performance only on that trait Principal components analysis (PCA) A factor analytical method of screening a large number of simultaneous (i.e. multivariate) measures for those which are best regarded as outcomes of a broader underlying factor because they vary together all or most of the time; Method to find new variables which have the highest possible variance and better represents the general differences between individuals than any other linear combination of the original variables Principle of evidence The philosophical idea that research provides evidence, not proof Principle of falsification Assertion that the scientific method is ultimately based on our ability to prove an assertion is false by finding a counterexample to it, not to prove one is true Prior learning assessment

Assessment Glossary


130

Techniques to assess student understanding and recall of material learned in previous, related courses, so that faculty can teach accordingly; Helps faculty determine the most effective starting point for a given lesson and the most appropriate level at which to begin instruction Priority goals Most important aims that a teacher or evaluator is trying to accomplish; An important aim or purpose of instruction Privacy rights A learner/teacher's privilege to have his/her performance on an assessment or evaluation results to be confidential and not disclosed to unauthorized parties without prior permission or consent Pro forma A term used to describe the presentation of data, usually financial statements, where the data reflect information as if the state of the world were different from that which is, in fact, the case (e.g. projected profit and loss statements given one or more projected or alternative levels of revenue) Proactive Acting in advance to deal with an expected difficulty; Behavior which seeks to find opportunities in order to take advantage of them and to anticipate threats in order to neutralize them Probabilistic Stating what is likely to occur, not necessarily what will occur Probabilistic cause A cause that usually produces an outcome such that changes in variable A tend to produce changes in variable B Probability The chance that a phenomenon will occur randomly; A term used in statistics that depicts the likelihood of some event occurring; See p-value Probability proportional to size A type of two-stage cluster sampling where each cluster’s chance of being selected in stage one depends on its population size Probability sampling Every individual in a specified population has a known probability of selection Probability value The probability of the result of your research study, or an even more extreme result, assuming that the null hypothesis is true Probes Prompts to obtain response clarity or additional information Problem-based learning The uses of simulated anchors, scaffolding, and working on related projects, along with opportunities to engage in tasks, as a way to facilitate student learning Problem of induction Things that happened in the past might not happen in the future

Assessment Glossary


131

Problem solving skills Defining the problem, being able to obtain background knowledge, generating possible solutions, identifying and evaluating constraints, choosing a solution, functioning within a problem solving group, evaluating the process, and exhibiting problem solving dispositions; Problem solving and Critical thinking skills overlap Procedures The directions for implementing all aspects of an assessment or evaluation process in accordance with the rules and guidelines given in an institution's policy; Specifications of how the assessment or evaluation is to be conducted, designated timelines, persons responsible, forms to be used, documentation to be provided, the analysis plan, and the steps to be followed Process How inputs are converted to outputs within an educational or other production system through linked activities; A generalizable method of doing something, generally involving steps or operations which are usually ordered and/or interdependent; How you decide to accomplish some task; How something is done; A process can be evaluated as part of an assessment, but the assessment process primarily focuses on products instead of processes Process evaluation Investigates issues regarding the program’s current operations or the implementation of new initiatives; Usually focuses on what the program does, who does it, and how it is done Process reengineering The process of improving business practices or methods by creating and implementing new processes or making changes to existing processes Process skills or process assessment Assessing a student's skills in progressing through a series of actions or operations; Process skills that teachers seek to assess relate to thinking abilities, applications of procedural knowledge, and interactions with others; For example: Critical thinking, creative thinking, problem solving, decision making, goal setting, cooperation, relating to others, leadership, and management Process standards Statements that describe skills students should develop to enhance the process of learning; Process standards are not specific to a particular discipline, but are generic skills that are applicable to any discipline Process variables, measures or indicators The manner in which teaching is conducted; Includes instructional strategies, sequencing of curricular content and skills, behavior management techniques, assessment and monitoring practices, and the use of materials and equipment; Indicators of instructional good practice that have been shown in research to have a reliable impact on educational outcomes but often neglected in outcomes assessment; Many faculty have difficulty relating to outcomes measures but are intuitively intrigued by process measures Processes A series of actions or operations leading toward a particular result; In education, the processes of learning, seasoning of character, creativity at the edges of knowledge and imagination, and responsible advocacy of values are considered critically important Proctor An individual who supervises a written examination or test to maintain a fair and consistent testing environment, but takes no part in the examination process Procurement budget

Assessment Glossary


132

A budget that displays the timing, quantities, and projected costs for each type of raw material required to meet inventory and production needs Producer price index (PPI) An inflation indicator issued by the U.S. Bureau of Labor Statistics used to assess the economy’s wholesale price levels Product The tangible and stable result of a performance or task Product or project assessment Assessment of the product which results from a project assigned to an individual or group based on a topic related to the curriculum Product differentiation A marketing strategy that attempts to make clear distinctions between comparable products serving the same market segment through the use of innovative design, packaging, branding, positioning, etc. Product life cycle A marketing concept in which company products or brands (including educational institutions and programs) follow a series of phases: introduction, growth, maturity and decline Productivity The amount of education, research, and training within a university, as a function of resources and the needs of students, parents, and taxpayers; How much is accomplished given the cost of accomplishing it Productivity measure A performance measure that quantifies how much product is produced at what cost, and that measures economy; A measure of need fulfillment or goal achievement relative to its cost Professional development A process designed to improve specific professional competencies or the overall competence of a teacher or other professional Proficiency State of having or demonstrating a high degree of knowledge or skill in a particular area; Sufficient expertise in a knowledge area or adequate mastery of a skill with regard to a standard; Performing in a given art, skill, or branch of learning with correctness and facility; Achieving competency on predetermined standard Proficiency level The equivalent of a cut score (on a forced-choice assessment) but for a performance/complex assessment; Set by determining the required performance criteria (such as the required level on a rubric) for a specific grade level; For example: A proficiency level could be the achievement of all the criteria required for a scoring level, or it could be a set number of points achieved by combining scores for each feature on a rubric Profile A representation of a learner/teacher's performance on a number of attributes, measures, or dimensions that use the same scale; A graphic compilation of the performance of an individual on a series of assessments; A graphic representation of an individual's scores (or their relative magnitudes) on several tests (or subtests) that employ a single standard scale Profit The return received on a business enterprise after all of its operating expenses and financial obligations have been met

Assessment Glossary


133

Profit and loss statement (P&L) An official financial statement prepared on a quarterly or annual basis showing earnings, expenses and net profit; A financial document summarizing company revenue and expenses during a specified period of time; A company’s net profit is determined from this financial report by subtracting total expenses from total revenue Profit budget A set of estimated financial statements and schedules for the following year Profit center An enterprise that has a manager who is responsible for profit performance and who controls most of the factors affecting revenues and costs Profit margin or margin Company earnings expressed as a percentage of revenue Profitability A common corporate objective that focuses on a company’s or industry’s efficiency at generating income Progeny quality of life The tendency for children of college-educated parents to graduate from high school, attend college, and have higher cognitive development; Also the tendency for daughters of college-educated mothers to avoid unmarried teen pregnancy Program A set of courses which culminates in a degree, certificate, or preparation for degree or certificate; A plan or system under which action may be taken towards a goal; A set of courses constituting an area of specialization; A coherent set of learning and developmental objectives Program accreditation or specialized program accreditation A process of program, unit, or discipline review where the examination of effectiveness is conducted within the context, requirements, and standards of a discipline-based accrediting body Program assessment Assessment to determine the extent to which students in a departmental program can demonstrate the learning outcomes for the program Program budgeting A budgeting method that deals mainly with broad planning and the associated costs of functions, activities, programs and services Program completion level A measure of progress, expressed as a percentage, toward completion of a program of studies; Derived by dividing a student's total number of formal award units previously earned and allowed by the institution, by the total number of units needed for completion of his/her program Program evaluations Evaluations that assess ongoing activities which provide services Program improvement Increasing the likelihood of achieving program objectives by using information acquired through assessment Program review or academic program review

Assessment Glossary


134

The process of evaluating the quality and effectiveness of a program; A process of self-study, evaluation, and planning completed by each academic and service area program on a regular cyclical basis; A systematic process of data collection, analysis, and interpretation for effective planning and accreditation review; An analysis of a program’s performance with respect to particular indicators, including student learning outcomes; In higher education, a program review could consist of anything from a judgment by experts of the worth of the curriculum based primarily on observations to a comparison between student performance indicators and objectives and actual student performance on either standardized or evaluator-constructed measures Program SLO or student learning outcome What the student will be able to produce at the end of a program; This is the middle level at which SLOs are usually assessed; Writing program-level SLOs involves considering the overarching missions of the respective program, matching these missions to courses in the program, and coming up with a cumulative assessment which may or may not be the same as a course-level assessment; Program-level SLOs, like core competencies, may also serve as “folders” or “categories” under which course-level SLOs are assessed; Program-level SLOs may or may not be directly assessed Progress measures Specific performance targets that impact the entire organization, address its strategic goals, are stated in terms of measurable and verifiable outcomes, and challenge the organization to be more responsive to the environment to achieve its desired future Project A form of complex performance assessment involving several types of activities and products for completion; A complex assignment involving more than one type of activity and production; Most projects involve planning and usually end with a report (oral or written) or product Project budgeting A budget that is developed around a specific type of project rather than time period Project evaluation Evaluations that assess activities that are funded for a defined period of time to perform a specified task; Examples include a three-day workshop on behavioral objectives, a two-year development effort, or a three-year career education demonstration Project logic A model that arrays the resources, activities and goals of a project to allow the relationships to be clearly viewed and understood Projection A prediction or an estimate of a future occurrence, based on current data or trends Projective measures A test-taking method in which the participants provide responses to ambiguous stimuli Projective techniques Psychological techniques often used in focus groups where participants are asked to make imaginative comparisons Prompt A verbal statement or question that provides a cue, reminder, or inspiration, or that motivates the learner/teacher being assessed into action; A statement to which a student responds in an assessment; The directions that ask the student to undertake a task Proportional stratified sampling

Assessment Glossary


135

Type of stratified sampling in which the sample proportions are made to be the same as the population proportions on the stratification variable Proprietary information Information associated with company products, business, or activities, including such items as financial data, trade secrets, product research and development, product designs, marketing plans or techniques, computer programs, processes, and know-how that has been clearly identified and communicated by the company as proprietary, a trade secret, or confidential Proprietary technology Technology such as software applications that is unique and legally owned by a single enterprise or individual Proprietary test or survey Measurement instrument owned by a company or individual Propriety The extent to which an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation as well as those affected by its results; The extent to which the evaluation has been conducted in a manner that evidences uncompromising adherence to the highest principles and ideals (including professional ethics, civil law, moral code, and contractual agreements) Prospective study Another term applied to a panel study Protocol The rules and formalities that guide the administration and scoring of an assessment and the implementation of an evaluation; A record or document of evidence and information relating to an assessment or evaluation Psychological attributes Traits that characterize an individual's behavior Psychometric Properties of the items and test such as the distribution of item difficulty and discrimination indices Psychometric analysis The analysis of the items and test such as the distribution of item, difficulty, and discrimination indices Psychometrician A qualified person who analyses the psychometrics of a test or item Psychomotor domain The range of locomotor behaviors needed to explore the environment and perform tasks as well as the sensory-motor activities that are essential to learning and communication Psychosocial Of or pertaining to the interaction between social and psychological factors Public domain Materials such as literary works, music, films, photographs, art, etc. that are no longer covered under copyright law and can be published without obtaining permission or paying a fee Public social benefits

Assessment Glossary


136

The advantages of postsecondary education that accrue to society in general that are not related to economic, fiscal, or labor market effects Public speaking One-to-many communication in a face-to-face situation Publish To release and make public, in hardcopy, electronic, or web-based formats, an assessment to make it widely available Published test A test that is publicly available because it has been copyrighted and published commercially Purchasing power The measure of the value of a particular currency expressed in terms of the amount of goods or services that one unit of currency can purchase Purpose The primary reason or intended use that provides direction for the design, interpretation, and use of an assessment and evaluation system; The objectives of an evaluation and the intended use of its reports Purpose of a research study A statement of the researcher’s intent or objective of the study Push polling A form of pseudo-research intended to change opinions rather than measure them, often by asking leading questions Purposive sampling The researcher specifies the characteristics of the population of interest and locates individuals with those characteristics

Assessment Glossary


137

Q Qualitative assessment Collection of data that does not lend itself to quantitative methods but rather to interpretive criteria; Methods that rely on descriptions rather than numbers: For example: Ethnographic field studies, logs, journals, participant observation, and open-ended questions on interviews and surveys Qualitative evidence, data, or information Data relating to, or involving quality or kind, which cannot be represented numerically; Data or evidence that is narrative or descriptive in form; Information presented and/or summarized in narrative form such as written expressions descriptive of a behavior or product; Facts and claims presented in narrative, not numerical, form; The facts and evidence that describe a learner/teacher's performance and that typically are recorded in written, audio, or visual form; Qualitative evidence usually involves fewer cases than quantitative data, but shows much more specific information and may be very subjective; For example: Portfolios of work, narrative description and evaluation of a performance, learner description and analysis of a learning experience Qualitative methods Ways to dig deeply into a question, usually related to words rather than numbers, and with evolving theories using small sample sizes; Methods that rely on descriptions rather than numbers (e.g. ethnographic field studies, logs, journals, participant observation, and open-ended questions on interviews and surveys); The metrics used to measure value through the use of means other than numerical factors to quantify accomplishments, progress, shortfalls, or specific results Qualitative observation Observing all potentially relevant phenomena Qualitative research Empirical research in which the researcher explores relationships using textual, rather than quantitative data; Case studies, observation, and ethnography are considered forms of qualitative research; Results of qualitative research are usually not considered generalizable, but are often transferable; Data typically in narrative form; Data gathered by use of observations and interviews; Results are usually contextual or unique to an individual and setting; Research in which questions are open-ended and results are expressed in non-numerical terms; Research that produces findings not arrived at by means of statistical procedures or other means of quantification; Qualitative data can be collected through interviews, focus groups, observation, or review of documents Qualitative research question An interrogative sentence that asks a question about some process, issue, or phenomenon to be explored Qualitative or nominal variable A variable whose values can be from some finite set of different possibilities Qualitizing Converting quantitative data into qualitative data Quality The degree to which a service meets the needs of the constituent; The degree to which an institution of higher education is fitted to its purpose, satisfactory to the clients, and of a quality grade equivalent to other institutions of higher education Quality assurance

Assessment Glossary


138

Activities or programs the purpose of which is to demonstrate and ensure that products and services meet specifications and are consistently of high quality Quality audit The process of examining the elements of a quality management system in order to evaluate how well they comply with quality system specifications Quality check The process of verifying both the accuracy of specific data and information, and the appropriateness of the techniques used to collect, score or rate, and to analyze the data and information Quality control Activities or programs whose purpose is to ensure that all quality specifications for products or services are being met and are of consistently high quality Quality improvement Any system or process designed to enhance an organization’s ability to meet quality requirements Quality program indicator A variable reflecting effective and efficient program performance Quantitative assessment Collection of data that can be analyzed using quantitative methods (methods that rely on numerical scores or ratings) Quantitative evidence, data, or information Data or evidence that is numerical in form; Data which can be represented numerically; Information presented and/or summarized in numerical form; Facts and claims that are represented by numbers; The facts and evidence that describe a teacher's performance and that typically are recorded in numeric, statistical, or graphic form, or can be meaningfully represented by numbers; Quantitative evidence usually involves a great number of cases and is used to show general patterns and trends rather than specifics and tends to be much more objective than qualitative evidence Quantitative methods Ways to collection great amounts of information at a more shallow level, usually related to numbers over words, testing of assumptions, and aggregate information with large sample sizes; Methods that rely on numerical scores or ratings Quantitative observation Standardized observation Quantitative research Empirical research in which the researcher explores relationships using numeric data; Results can often be generalized, though this is not always the case; Data is usually numerical and gathered by quantifying observations through administering tests or other instruments; Results are usually generalizable or an attempt to find laws and generalizations; Research that produces data that can be statistically analyzed and whose results can be expressed numerically Quantitative research question An interrogative sentence that asks a question about the relationship that exists between two or more variables Quantitative skills

Assessment Glossary


139

The ability to apply arithmetic operations, either alone or sequentially, using numbers embedded in printed materials to solve a problem Quantitative theory A theory in which terms are expressed numerically rather than verbally; Quantitative theory states that reality is objective, singular, and apart from the observer and that research is a deductive process with context-free cause and effect relationships; Generalizations from these relationships lead to prediction, explanation, and understanding Quantitative or interval variable A variable whose values are numbers; A variable that varies in degree or amount Quantitizing Converting qualitative data into quantitative data Quartile The 3 points on a percentile scale which divide the scale into equal quarters; One of three percentiles in a frequency distribution used to define one of four equivalent components; The breakdown of an aggregate of percentile rankings into four categories: The 1st quartile is the 25th percentile, the 2nd quartile is the 50th percentile or median, and the 3rd quartile is the 75th percentile Quasi-experiment or quasi-experimental research design An experimental research design that does not provide for full control of potential confounding variables primarily because it does not randomly assign participants to comparison groups; Similar to true experiments but uses nonrandomized groups; Incorporates interpretation and transferability in order to compensate for lack of control of variables; An investigation that has all the elements of an experiment, except that subjects are not randomly assigned to groups; Common in field research and program evaluations where, unlike true experimental designs, research subjects cannot be randomly assigned to their respective grouping in the study (for actual or ethical reasons) Questionnaire Written documents that contain a series of questions that are answered by respondents; One or more questions presented and answered together; A list of questions concerning a particular topic administered to a group of individuals to obtain information concerning their preferences, beliefs, interests, and behavior; An instrument consisting of a series of queries and statements that is used to collect data and information from a learner/teacher; A self-report data collection instrument filled out by research participants Quick ratio The measure of a company’s liquid assets to current debts used to determine the company’s ability to meet its financial obligations; Calculated by subtracting inventories from current assets and then dividing the results by current liabilities Quintile The breakdown of an aggregate of percentile rankings into five categories: the 0-20th percentile, 21-40th percentile, 41-60th percentile, 61-80th percentile, and 81-100th percentile Quixotic reliability Situation where a single manner of observation consistently, yet erroneously, yields the same result Quota sampling The researcher determines the appropriate sample sizes or a quota for the groups identified as important and takes convenience samples from those groups

Assessment Glossary


140

R Random Affected by chance Random assignment A procedure that makes assignments to conditions on the basis of chance and in this way maximizes the probability that the comparison groups will be equated on all extraneous variables Random error Any unsystematic error; A quantity (often observed indirectly) that appears to have no relationship to any other variable Random sampling or sample In research and statistical studies, a random sample is one where the researcher ensures that each member of that population has an equal chance of being selected; Process used in research to draw a sample of a population strictly by chance, yielding no discernible pattern beyond chance; Drawing a number of individuals from a larger group or population, so that all individuals in the population have the same chance of being selected; Drawing a number of items of any sort from a larger group or population so that every individual item has a specified probability to be chosen; Random sampling can be accomplished by first numbering the population, then selecting the sample according to a table of random numbers or using a random-number computer generator; The sample is said to be random because there is no regular or discernible pattern or order; Random sample selection is used under the assumption that sufficiently large samples assigned randomly will exhibit a distribution comparable to that of the population from which the sample is drawn; Sample selection according to a random process, with the selection of each entity in no way dependent on the selection of other entities Random selection Randomly selecting a group of people from a population Randomization Used to allocate subjects to experimental and control groups; The subjects are initially considered not unequal because they were randomly selected Randomized controlled trial (RCT) Robust and well-tried research design for evidence-based practice where participants are drawn at random from the largest practicable pool (i.e. they are not pre-selected subjects who are in some way likely to fit the treatment being evaluated), and includes control groups who do not receive the treatment in question but who in every other respect are treated identically to the treatment group (to insure against making Type I errors should a variable other than the treatment be responsible for changes in the treatment group); RCTs are also expected to be blind or double-blind where necessary, to generally avoid bias and confounding, to do their best to design out practice effects, order effects, fatigue effects, and ceiling effects, and to maximize (or at least quantify) the many subtypes of research validity and reliability Randomizing A method of picking assessment items and presenting them in no particular order to reduce the likelihood of cheating Range The extent of variation in the measurement of a variable; A statistical variation measure based on the distance between the highest and lowest score in a series of scores; Difference between the smallest and the largest

Assessment Glossary


141

value of a measure; Difference between the highest and lowest scores in a distribution (e.g. if the highest score on a test is 95 and the lowest score is 45, the range is 50); Sometimes expressed as a formula (e.g. 95-45) or phrase (e.g., 95 to 45) Rank order A rating method where the performance of a group, process or product is arranged in a particular order, such as highest to lowest Ranking The process of ordering or arraying from highest to the lowest; The ordering of responses into ascending or descending order; A learner/teacher's level of performance is compared to others rather than being judged independently of how others perform Ranked or ordinal variable Variables that come in a definite order, so that you might make them into numbers (e.g. Excellent = 4, Good = 3, Fair = 2, Poor = 1), but it would not make sense to do arithmetic with them (e.g. Fair + Poor = Good) Rate To assign judgments or estimations to the magnitude of degree of some aspect of learning/teaching behavior or performance; The percentage of people in a group who have a specific characteristic Rate of return The return rate earned on an investment on an annual basis, expressed as a percentage Rater A person who evaluates or judges student performance on an assessment against specific criteria Rater drift The tendency for assessors and evaluators to unintentionally redefine criteria and standards over time or across a series of ratings Rater effect The tendency of an assessor or an evaluator to rate a performance at a level that does not accurately or consistently reflect the performance level of that learner/teacher; There are several types of rater effect, all of which are possible sources of systematic measurement error Rater training Process of educating raters to evaluate student work and produce dependable or reliable scores; Typically uses anchors to acquaint raters with criteria and scoring rubrics, open discussions between raters and the trainer help to clarify scoring criteria and performance standards, and opportunities for raters to practice applying the rubric to student work; Often includes an assessment of rater reliability that raters must pass in order to score actual student work Rating A systematic estimation of the magnitude or degree of some attribute of learning/teaching, using a numerical or descriptive continuum Rating scale A scale based on descriptive words or phrases that indicate performance levels; Scale in which qualities of a performance are described (e.g. advanced, intermediate, novice) in order to designate a level of achievement; May be used with rubrics or descriptions of each level of performance; Subjective assessments are made on predetermined criteria for documenting where learners fall on a continuum of proficiency; A continuum of response choices; Subjective assessments made on predetermined criteria in the form of a scale; Rating scales

Assessment Glossary


142

include numerical scales or descriptive scales; Forced choice rating scales require that the rater determine whether an individual demonstrates more of one trait than another Ratio analysis Use of ratios to measure a company’s financial performance by conveying the relationships between an organization’s accounting numbers and their trends over time; Used to establish values and evaluate risks Ratio scale A scale of measurement that has a true zero point as well as the characteristics of the nominal (labeling), ordinal (rank ordering), and interval (equal distance) scales Rational model of decision making Derived from economic theories of how to make the "best" decisions; Involves efforts to move toward consciously-held goals in a way that requires the smallest input of scarce resources; Assumes the ability to separate ends from means, rank all alternatives, gather all possible data, and objectively weigh alternatives; Stresses economic rationality in the process of reaching decisions Rationale Written statements providing the reasons for steps taken and choices made Rationalism The philosophical idea that reason is the primary source of knowledge Raw score The measure prior to scaling; A score obtained from a test, assessment, observation, or survey that has not been converted to another type of score such as a standard score, percentile rate, ranking, or grade; The number of items that are answered correctly out of the total possible; By itself, a raw score provides little useful information about an individuals' performance; Examples of raw scores include a count of the number of correct answers on a vocabulary text, a tabulation of the occurrence of a certain type of event during an observation, or an initial rating on a portfolio document Reactence Type of possible confounding where stubbornness on the part of subjects will result in responses to the research instrument that are deliberately opposite to those which might otherwise have been made Reactivity An alteration in performance that occurs in people as a result of being aware of participating in a study Readiness test A test of ability to engage in a new type of specific learning; Level of maturity, previous experience, and emotional and mental set are important determinants of readiness Reading skills Locating, understanding, and interpreting written information in documents such as manuals, graphs, and schedules Repeated measures design Type of experimental design where the scores from two or more different groups of subjects are measured on two or more occasions and compared using a test for the difference of means (e.g. in testing the hypothesis that sober subjects are better at mathematics than drunk ones, a repeated measures design would test the same subjects on different occasions when they are drunk and when they are sober) Reasonableness

Assessment Glossary


143

A characteristic of scoring criteria ensuring that the judging of student work is appropriate for the conditions within which the task was completed Reasoning The process by which one is motivated to, and looks for, evidence to support and refute a statement or proposition Reception analysis A type of audience research that focuses on what audiences perceive in the media - as opposed to what broadcasters think they produce Recommendations A set of suggestions derived from the learner/teacher evaluation results; For formative teacher evaluation, they may include a list of professional development activities and a plan of assistance; For summative teacher evaluation, they may consist of personnel actions such as tenure, dismissal/termination, reassignment/transfer, contract renewal, or promotion Reconciliation The process of confirming that “checkbook” balances correspond to bank statements Record The written or taped data, evidence, judgments, notes, recommendations, and other statements for use in the teacher evaluation process; To register and store data and other information Reduction in force (RIF) Layoffs of employees necessitated by reductions in budgets or decreases in student enrollment; RIF decisions are frequently based on seniority rather than level of performance and staffing needs; An involuntary separation of an employee or groups of employees due to economic pressures, lack of work, organizational changes, or other reasons of business necessity that require a reduction in staff Redundancy payment Refers to an amount paid to an employee who has been laid off, typically calculated in relation to length of employment Reengineering The redesigning of business and work processes, policies, or organizational structure Reference group The norm group that is used to determine the percentile ranks Reference population The population of test takers represented by test norms; The sample on which test norms are based must permit accurate estimation of the test score distribution for the reference population; Reference populations may be defined in terms of examinee age, grade, or clinical status at the time of testing, or other characteristics Reflection The process by which a learner/teacher reviews his/her past performance as a means of improving future performance Reflective essays Generally brief (five to ten minute) essays on topics related to identified student learning objectives, although they may be longer when assigned as homework; Students are asked to reflect in writing on a selected issue, and content analysis is used to analyze the results

Assessment Glossary


144

Reflective practice A mode that integrates or links thought and action with reflection; Involves thinking about and critically analyzing one's actions with the goal of improving one's professional practice; Requires individuals to assume the perspective of an external observer in order to identify the assumptions and feelings underlying their practice and then to speculate about how these assumptions and feelings affect practice Reflexivity Self-reflection by the researcher on his or her biases and predispositions Regression analysis A set of statistical procedures used to explain or predict the values of a dependent variable on the basis of the values of one or more independent variables Regression artifacts, effect, or regression to the mean The tendency of very high scores to become lower and very low scores to become higher on posttesting of other or the original measure; The tendency of examinees scoring above or below the mean of a distribution on a pretest to score closer to the mean on the posttest; A statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated, resulting in the sample's posttest mean being closer to the posttest population mean than their pretest mean was to the pretest population mean; A statistical phenomenon that can make natural variation in repeated data look like real change Regression coefficient A multiplier of an independent variable in a linear equation relating a dependent variable to a set of independent variables; The coefficient is said to be standardized or under-standardized accordingly as the variable it multiples has been scaled to a standard deviation of 1.0 or has some other standard deviation; The predicted change in Y given a one unit change in X; When the regression line is linear (y = ax + b) the regression coefficient is the constant (a) that represents the rate of change of one variable (y) as a function of changes in the other (x); The slope of the regression line Regression-discontinuity design A design that assesses the effect of a treatment condition by looking for a discontinuity in regression lines between individuals who score lower and higher than some predetermined cutoff score Regression effect See Regression artifacts Regression equation The equation that defines the regression line Regulation A detailed set of orders that clarify and implement a particular section of law; Government activity designed to monitor and guide private economic competition; Specific actions (characterized as economic regulation) have included placing limits on producers' prices and practices, and promoting commerce through grants or subsidies; Other actions (termed social regulation) have included regulating conditions under which goods and services are produced and attempting to minimize product hazards and risks to consumers Regulatory policy Policy designed to limit the actions of persons or groups in order to protect all or parts of the general public Relative decision Decisions based on the use of the standing of individuals relative to one another Reinforce

Assessment Glossary


145

To strengthen a learned way of behaving by some response-contingent external or internal influence Relative score interpretations The meaning of the score for an individual, or the average score for a definable group, derived from the rank of the score or average within one or more reference distributions of scores Relevance The extent to which the domains and indicators covered by a teacher evaluation system apply to a teacher's professional functions in terms of both importance and frequency Reliability The consistency or stability of test scores; Degree to which an assessment yields dependable and consistent results; The measure of consistency for an assessment instrument; Instruments should yield similar results over time with similar populations in similar circumstances; The degree to which test scores for a group of test takers are consistent over repeated applications of a measurement procedure and hence are inferred to be dependable, and repeatable for an individual test taker; The degree to which scores are free of errors of measurement for a given group; A measure of how consistent the results obtained in an assessment are in a norm-referenced evaluation situation; Consistency of a student's ranking within the group of students against which the student is being compared; The degree to which an assessment or instrument consistently measures an attribute; Extent to which an experiment, test, or any measuring procedure yields the same result on repeated trials; Degree to which the results of an assessment are dependable and consistently measure particular student knowledge and/or skills; When the same results are obtained over time, the way of collecting the results are considered reliable; For example: Repeated measurements show the same achievement or several observers of a classroom situation closely agree with ratings recorded for individuals on the same criterion; There are several types of reliabilities, including Intra-Rater (the degree to which the measure yields consistent results over different administrations with the same teacher performing at the same level by the same assessor), Inter-Rater (the degree to which the measure yields similar results for the same teacher at the same time with more than one assessor), Internal Consistency (the degree to which individual observations or items consistently measure the same attribute), and Test-Retest (the degree to which the measure produces consistent results over several administrations assessing the same attribute of a teacher) Reliability coefficient A unit-free index that reflects the degree to which scores are free of measurement error; Sometimes a product-moment correlation; In classical test theory, the term represents the ratio of true score variance to observed score variance for a particular examinee population; A calculated number whose value must be between 0 and 1 which describes the consistency of the assessment results such that the larger the number's magnitude, the more consistent the assessment (e.g. if the coefficient value were 1, all students' scores would be expected to rank exactly the same way on retesting); A coefficient of correlation between two administrations of a test; The conditions of testing administration may involve variations in test forms, raters or scorers, or the passage of time that give rise to qualifying adjectives being used to describe the particular coefficient (e.g. parallel form reliability, rater reliability, and test-retest reliability); See Reliability estimates Reliability estimates There are four general classes of reliability estimates, each of which estimates reliability in a different way: Inter-rater or Inter-observer reliability (used to assess the degree to which different raters or observers give consistent estimates of the same phenomenon), Test-Retest reliability (used to assess the consistency of a measure from one time to another), Parallel forms reliability (used to assess the consistency of the results of two tests constructed in the same way from the same content domain), and Internal consistency or Item reliability (used to assess the consistency of results across items within a test); see also Reliability coefficient Reliable measures A measure that provides consistent indications of the characteristics being measured; Measures that produce consistent responses over time Remediation

Assessment Glossary


146

Those techniques or strategies designed to improve a learner/teacher's performance in general deficiencies or specific areas of weakness Repeated measures A single sample of individuals is tested more than once on the dependent (outcome) variable; The same subjects are used for every treatment Repeated-measures design A design in which all participants participate in all experimental treatment conditions Repeated sampling Drawing many or all possible samples from a population Replicable An attribute of an assessment, observation system, or evaluation indicating that the process used to obtain the data and evidence is explicit and can be repeated Replicate To repeat an evaluation with all essentials unchanged Replication Research examining the same variables with different people and, often, in slightly different ways Replication logic The idea that the more times a research finding is shown to be true with different sets of people, the more confidence we can place in the finding and in generalizing beyond the original participants Report card Summary of student achievement, either formative or summative, describing student progress and development with respect to learner expectations, cognitive ability, and expected behavior Reporting The process of communicating results and recommendations to the designated individuals or groups Reporting scheme A record of evidence describing a learner/teacher's performance; There are two major types of reporting schemes: Complex (a record of evidence describing performance that provides detailed information about the performance and the context) and Reductive (a record of evidence describing performance that simplifies the data collection through classifying, coding, or analyzing them) Representative sample A sample that resembles the population; A method of dividing a population into strata and then taking a random sampling from each subpopulation Representativeness A factor of performance tasks and of scoring criteria ensuring that the task and criteria focus on the significant elements, concepts, and strategies in the outcome(s) assessed Request for price (RFP) A request made by a private company or government agency to potential vendors or contractors regarding a price quote for specified products or services Request for proposal (RFP)

Assessment Glossary


147

A formal invitation containing a scope of work which seeks a formal response (proposal) describing both methodology and compensation to form the basis of a contract; A document an organization sends to a vendor inviting the vendor to submit a bid for a product or service Requisition A written request to purchase a good or service Research Studying, developing, or testing a theory or phenomena by gathering data in a systematic way; The principle method for acquiring knowledge and uncovering the causes for behavior Research based teacher evaluation (RBTE) Teacher evaluation approach based on "empirically-validated" criteria or indicators of competence derived from research studies of effective teaching practices Research design The outline, plan, or strategy used to answer a research question; Science and art of planning procedures for conducting studies so as to get the most valid findings; Set of instructions for gathering and interpreting evidence Research ethics A set of principles to guide and assist researchers in deciding which goals are most important and in reconciling conflicting values Research hypothesis The hypothesis of interest to the researcher and the one he or she would like to see supported by the study results Research in practice Research in practice includes reading and responding to research, reflecting on practice in light of research, applying research findings to practice, and doing research about practice; It includes a range of ways that practitioners might engage in research, but also explicitly identifies reading and reflection as important ways for practitioners to engage with research Research literature Set of published research studies on a particular topic Research method Overall research design and strategy Research paradigm or approach A perspective based on a set of assumptions, concepts, and values that are held by a community of researchers; The collective set of attitudes, values, beliefs, procedures, and techniques that create a framework of understanding through which theoretical explanations are formed Research plan The outline or plan that will be used in conducting the research study Research problem An education issue or problem within a broad topic area; An issue that lends itself to systematic investigation through research Research proposal

Assessment Glossary


148

A written document that summarizes the prior literature, identifies the research topic area and the research questions to be answered, and specifies the procedure that will be followed in obtaining an answer to these research questions Research purpose See Purpose of a research study Research question A clear statement in the form of a question about the specific issue that a researcher wishes to answer in order to address a research problem; Questions that ask what variables can and will be manipulated and studied; See also Quantitative research question and Qualitative research question Research topic The broad subject matter area to be investigated Researcher-as-detective Metaphor applied to the researcher when searching for cause and effect Researcher bias Tendency for a researcher to obtain results consistent with what he/she wants to find because the beliefs, values, or desires of the researcher are reflected in their choice of methodology or interpretation of findings Resource requirements A quantitative description (usually in money) of the capital, equipment, information, labor, and land that achievement of a tactical objective will require Respondent A person that participates in a survey process by answering questions Response Result of a trial with a particular treatment Response bias The tendency subjects have to produce experimental responses, which are socially desirable, or that they think the experimenter expects; A participant's tendency to respond in a particular way or style to items on a test that yields systematic, construct-irrelevant error in test scores Response rate The percentage of people in a sample that participate in a research study; In survey research, the actual percentage of questionnaires completed and returned Response set The tendency to respond in a specific direction regardless of content Responses The answers to test, interview, or questionnaire items Responsibility That which a person is expected and obligated to do and for which he/she is accountable Responsibility center An enterprise whose performance is evaluated separately and is held responsible for its contribution to the firm’s mission and goals; See cost and profit center

Assessment Glossary


149

Resource allocation The determination and allotment of resources or assets necessary to carry out strategies and achieve objectives, within a priority framework Resource plan A component of the program definition statement stating how the program will be resourced and specifying what supporting services, infrastructure, and third party services are required Resources A source of supply or support; In a university setting, resources include personnel, raw materials and supplies, and the services of capital goods Restriction in range or variability Reduction in the observed score variance of an examinee sample, compared to the variance of the entire examinee population, as a consequence of constraints on the process of sampling examinees Restructuring Changing an organizational structure in order to make it more efficient and cost effective Results Findings from a study; What has been learned by students, departments, or institutions; The consequences and outcomes of a process or an assessment; May be tangible such as products or scores, or intangible such as new understandings or changes in behavior; Outcomes or assessment data obtained about student learning or development; Frequencies obtained from performance indicators Retention The continuation of a student's involvement in an academic endeavor until graduation Retention rate Percent of students remaining within an institution once matriculating at that institution Retrenchment The act of reducing expenditures in order to attain financial stability Retrospective questions Questions asking people to recall something from an earlier time Retrospective research The researcher starts with the dependent variable and moves backward in time Return on assets (RA) An internal measure of a company’s profitability, equal to a fiscal year’s profits divided by its total assets, expressed as a percentage (net profit divided by total assets) Return on investment (ROI) A ratio of the benefit or profit derived from a specific investment compared to the cost of the investment itself Return on net assets The ratio, expressed as a percentage, of profit gains or asset increases realized by a company during a financial year Revenue

Assessment Glossary


150

The amount of income generated by a product or service over a period of time Revenue budget The estimated dollar value based on a projection of future sales of products or services Review To examine again or to look at thoroughly in order to assign a grade, make a judgment, come to a conclusion, or evaluate RIF See Reduction in force Right-to-know audience A group of people entitled to be informed about the results of an evaluation or assessment; See Stakeholders Rightsizing An approach to reducing staff whereby jobs are prioritized in order to identify and eliminate unnecessary work; This method uses a selection criteria based on individual jobs rather than people in order to avoid possibly laying off the wrong employees; Frequently misused by organizations when layouts are due to a simple reduction in force (RIF) without any job analysis prioritization Rigor Degree to which research methods are scrupulously and meticulously carried out in order to recognize important influences occurring in an experiment Risk analysis The process of trying to determine in advance any potential negative occurrences and resulting impact on business operations that such occurrences would constitute Risk avoidance Strategies and tactics designed and implemented to circumvent or eliminate potential financial and other risks Risk management The use of insurance and other strategies in an effort to minimize an organization’s exposure to liability in the event a loss or injury occurs Role expectations Source of two types of possible confounding, where respondents in the role of a "good subject" may try to respond in the way they think they ought to respond rather than how they would normally respond, and those in the "bad subject" role may try to respond in any way which is not how they would normally respond Root cause analysis A procedure for ascertaining and assessing the causes of operational problems to assist in determining causes and implementing prevention strategies Rubric or scoring guide A scoring scale used to evaluate student work composed of at least two criteria by which student work is to be judged on a particular task and at least two levels of performance for each criterion; Some of the definitions of rubric are contradictory; Generally, a rubric is a scoring guide used in subjective assessments; Implies that a rule defining the criteria of an assessment system is followed in evaluation; May be an explicit description of performance characteristics corresponding to a point on a rating scale; A scoring rubric makes explicit expected qualities of performance on a rating scale, or the definition of a single scoring point on a scale; A description of a specific level of performance within a performance scale; Scoring guidelines written and shared for judging performance that indicate the qualities by which levels of performance can be differentiated, and that anchor

Assessment Glossary


151

judgments about the degree of achievement; A kind of holistic or primary trait scoring in which detailed criteria are delineated and used to discriminate among levels of achievement in assignments, performances, or products; A generic scoring tool used to evaluate a student’s performance in a given outcome area, consisting of a fixed measurement scale and a list of criteria that describe the characteristics of products or performances for each score point on that scale; A brief statement organized in descending order of accomplishment describing a certain quantity or quality of work, learning, or behavior; Specific sets of criteria that clearly define for both student and teacher what a range of acceptable and unacceptable performance looks like; Criteria define descriptors of ability at each level of performance and assign values to each level; Levels referred to are proficiency levels which describe a continuum from excellent to unacceptable product; A set of primary traits and guidelines for scoring and evaluating each assessment as agreed upon by a particular faculty group; A rubric makes explicit and specific statements about the expected qualities of performance at each point in the scale and for each primary trait or standard being evaluated; Rubrics can be used to help ensure consistency among raters and increase inter-rater reliability; An objective evaluation tool using specific criteria; A set of categories that define and describe the important components of the work being completed, critiqued, or assessed; Each category within a rubric contains a graduation of levels of completion or competence with a score assigned to each level and a clear description of what criteria need to be met to attain the score at each level; A description of the criteria which will be used to grade or assess student work or activities and the grading scheme based on those criteria; Rubrics may be analytic (specifying many traits which will be evaluated, and with the score on each trait added together to produce the overall grade for the work or activity) or holistic (providing holistic descriptions of the type of work produced at each level Rule making Administrative establishment of general guidelines for application to a class of people or a class of actions at some future time Rule of parsimony Preferring the most simple theory that works

Assessment Glossary


152

S Sales budget A budget that displays a company’s projected sales by month, product type, and geographical area Salient or salience A striking point or feature; Something distinctively different or important Sample The set of elements taken from a larger population; A subset of cases drawn from a population; A part of a population; A subset of the entire, total, or parent population of scores; A sample is a group of units selected from a larger group (the population); A collection of individuals, drawn from the population of interest, generally to try to understand the population as a whole without having to examine every individual; Usually research is done with a sample, rather than the entire population, in the hope of drawing valid conclusions about the larger group or parent population; A selection of a specified number of entries called the sampling unit (participants, items, etc.) from a larger specified set of possible entities, called the population Sampling The process of drawing a sample from a population; A way to obtain information about a large group by examining a smaller, randomly chosen selection (the sample) of group members; When done correctly, the results will be representative of the group as a whole; May also refer to the choice of smaller tasks or processes that will be valid for making inferences about the student's performance in a larger domain; Matrix sampling asks different groups to take small segments of a test so that the results will reflect the ability of the larger group on a complete range of tasks Sampling bias A bias arising from logically flawed or carelessly executed sampling, resulting in a sample which does not fairly represent the population in question Sampling distribution The theoretical probability distribution of the values of a statistic that results when all possible random samples of a particular size are drawn from a population; The sampling distribution describes probabilities associated with a statistic when a random sample is drawn from a population; The sampling distribution is the probability distribution or probability density function of the statistic; Derivation of the sampling distribution is the first step in calculating a confidence interval or carrying out a hypothesis test for a parameter Sampling distribution of the mean The theoretical probability distribution of the means of all possible random samples of a particular size drawn from a population Sampling error The difference between a sample statistic and a population parameter; The degree to which the results from the sample deviate from those that would be obtained from the entire population because of random errors in the selection of respondents and the corresponding reduction in reliability; The deviation between the characteristics of a sample and a population Sampling frame A listing that should include all those in the population to be sampled and exclude all those who are not in the population; A list of all the elements in a population Sampling interval

Assessment Glossary


153

The population size divided by the desired sample size; Symbolized by “k” Sampling of performance The selection of an array of learning/teaching performances and settings to be evaluated so that they are wide enough in range and large enough in numbers to cover the scope of the performances addressed by the evaluation system in terms of representativeness and comprehensiveness, and to permit valid inferences about performances to be made Sampling with replacement Sampling technique where it is possible for elements to be selected more than once Sampling without replacement Sampling technique where it is not possible for elements to be selected more than once Sanction A penalty for poor performance; Something given to encourage improvement when performance or service levels are not satisfactory Sarbanes-Oxley Act The Sarbanes-Oxley Act of 2002 was enacted to increase the accountability of organizations to their stakeholders; Two provisions are of special interest to HR professionals are the whistleblower protection provision and the 401(k) blackout notice provision Satisficing The tendency to select the first option that meets a given need or to select the option that seems to address most needs rather than the “optimal” solution; Critique of the economics model of profit maximization used in rational decision making theory which proposes that people in real world decision making situations use a form of bounded rationality and actually seek solutions which are only sufficiently satisfying (i.e. satisficing); A limited search through familiar patterns of behavior for a course of action that meets pre-established, minimum standards of performance, rather than an exhaustive review of all alternatives in search of the optimal solution to a particular problem; A decision making strategy that attempts to meet criteria for adequacy, rather than to identify an optimal or maximizing solution; The process of decision making that characterizes most governmental action Satisfaction A student's attitude towards their educational environment and its components Scalability The degree to which a computer application or component can be expanded in size, volume, or number of users served and continue to function properly Scale A classification tool or counting system designed to indicate and measure the degree to which an event or behavior has occurred; The system of numbers, and their units, by which a value is reported on some dimension of measurement; In testing, scale sometimes refers to the set of items or subtests used in the measurement and is distinguished from a test in the type of characteristic being measured; A standard of measurement using equal intervals and allowing progressive classification (e.g. a thermometer or a Likert Scale); A series of numerical or descriptive ratings on a continuum used to assess or judge specific levels of performance; The following are the types of scales of measurement: Nominal scale: Naming used to label, classify, or categorize data (e.g. gender, Social Security Number, number on an athletic jersey, locker number, and an address); Ordinal scale: Classification function plus observations are ordered but the distance between adjacent values not necessarily the same (e.g. Olympic medals, finishing place in a race, and a class ranking); Interval scale: Classification, plus ordered, plus equal intervals between adjacent units (e.g. all test scores are assumed to be at an interval scale, temperature Fahrenheit, and temperature Centigrade); Ratio scale: All of the above plus

Assessment Glossary


154

the scale has an absolute and meaningful zero; Most physical measures are at the ratio level of measurement (e.g. height, weight, distance, time, pressure, and temperature on the Kelvin scale) Scale scores Scores on a single scale with intervals of equal size; A score to which raw scores are converted by numerical transformation (e.g. conversion of raw scores to percentile ranks or standard scores); Units of a single, equal-interval scale that are applied across levels of a test; The scale can be applied to all groups taking a given test, regardless of group characteristics or time of year, making it possible to compare scores from different groups of examinees; Scores appropriate for various statistical purposes since they can be added, subtracted, and averaged across test levels; Scores based on a scale ranging from 001 to 999; Useful in comparing performance in one subject area across classes, schools, districts, and other large populations, especially in monitoring change over time Scaling The process of creating a scale or a scaled score; Scaling may enhance test score interpretation by placing scores from different tests or test forms onto a common scale or by producing scale scores designed to support criterion-referenced or norm-referenced score interpretations SCANS (Secretary's Commission on Necessary Skills) An initiative of the United States Department of Labor that identified the skills workers need to perform in the world of work and which made recommendations for changes in secondary education to facilitate the development of these skills; The SCANS report was published in 1991 and identified five competencies (allocating resources, working with others, using information, understanding systems, and using technology) and three foundational skill sets (basic skills, thinking skills, and personal qualities) SCANS competencies Five essential requirements for students going directly to work and those planning further education: resources, interpersonal, information, systems, technology SCANS skills Three foundational skills sets identified by the United States Department of Labor as needed for effective job performance today: basic skills, thinking skills, personal qualities Scatter plot A graph used to depict the relationship between two quantitative variables Scenario planning The process of identifying economic, social, political and technological trends and exploring the implications of projecting them forward Schedule Part of the procedures of an evaluation/assessment; The designated dates and times for various activities related to an evaluation/assessment, the location of such activities, and the people who will be involved School profile A description (graphic, numeric, or variable) of the status of a school with respect to a set of concepts or variables Scientific management A formal theory of organization developed by Frederick Taylor in the early 1900s; Concerned with achieving efficiency in production, rational work procedures, maximum productivity, and profit; Focuses on management's responsibilities and on "scientifically" developed work procedures, based on "time and motion" studies Scope of content

Assessment Glossary


155

The extent of coverage, by an instrument or process, of all knowledge, skills, abilities, and behaviors to be measured, in terms of both breadth and depth of coverage Score A rating of performance based on a scale or classification; A specific value in a range of possible values describing the performance of a student; The number of points earned on a measure or the degree of success on an assessment of teacher attributes or performance; The result obtained by a student on an assessment, expressed as a number; Scores are usually expressed in numerical terms, but sometimes in descriptive terms or graphically Scorer An assessor who summarizes the results of an assessment for use by an evaluator or decision maker Scoring The process of determining the value of a performance on an indicator or criterion; A package of guidelines intended for people scoring performance assessments; May include instructions for raters, notes on training raters, rating scales, samples of student work exemplifying various levels of performance Scoring criteria Rules for assigning a score or the dimensions of proficiency in performance used to describe a student's response to a task; May include rating scales, checklists, answer keys, and other scoring tools; In a subjective assessment situation, a rubric Scoring dimension An attribute or facet of behavior or performance in a domain; Usually determined through logical or statistical analysis, and sometimes reported as part scores Scoring key A list of correct answers for selection items or the scoring guide to be followed in scoring or judging responses to constructed response items Scoring protocol The established criteria, including rules, principles, and illustrations, used in scoring responses to individual items and clusters of items; Usually refers to the scoring procedures for assessment tasks that do not provide enumerated responses from which test-takers make a choice Scoring rubric A set of rules, guidelines, or benchmarks at different levels of performance, or prescribed descriptors for use in quantifying measures of learner/teacher attributes and performance; A set of related scoring scales used for judging student work and awarding points to reflect the evaluation of the work; The principles, rules, and standards used in scoring an examinee performance, product, or constructed response to a test item; May vary in the degree of judgment entailed, in the number of distinct score levels defined, in the latitude given scorers for assigning intermediate or fractional score values, and in other ways Scoring scale Assessment criteria formatted as levels of quality ranging from poorest to best, used to judge student work on a single feature, and may combine several traits within a feature; Scoring levels on the scale are assigned points, each level specifying the characteristics of the quality of content or skills needed to attain the points Screening or screening test A fast, efficient measurement for a large population to identify individuals who may deviate in a specified area, such as the incidence of maladjustment or readiness for academic work; A test that is used to make broad categorizations of examinees as a first step in selection decisions or diagnostic processes

Assessment Glossary


156

Scripting The writing of evidence and notes throughout an assessment activity (e.g. interview, classroom observation) about what is happening; Does not include making judgments or interpreting the evidence and notes; May be continuous throughout the activity or intermittent at pre-specified time intervals (e.g. 3 minutes on, 2 minutes off) Search conference A qualitative research technique where a large group of people meets to thrash out an issue which is often used in community planning Secondary analysis The re-analysis of already existing quantitative data rather than text or verbal data; Often involves combining information from multiple databases to examine research questions Secondary data Aggregate information related to students, departments, or other groups; Also known as indirect data; Existing data originally collected or left behind at an earlier time by a different person for a different purpose Secondary data analysis A reanalysis of data using the same or other appropriate procedures to verify the accuracy of the results of the initial analysis or for answering different questions Secondary source A source that was created from primary sources, secondary sources, or some combination of the two Secondary standards Those standards that apply to the evaluation process and assessment methods rather than to the learner/teachers being evaluated or to their performance levels; Standards for which it is desirable, but not crucial or important, that they be met or addressed before the assessments are administered and the evaluation process is implemented Segmenting Dividing data into meaningful analytical units Selected response item or format A question or incomplete statement that is followed by answer choices, one of which is the correct or best answer; Also referred to as multiple-choice; Format which presents alternative responses, from which the student chooses the correct or preferred answer; An exercise for which examinees must choose a response from an enumerated set (multiple choice) rather than create their own responses or products (performance assessment) Selection bias Bias that occurs when the method of selecting study participants is skewed, resulting in participants coming from a particular social or economic group, and not from others Selection-history effect Occurs when an event occurring between the pretest and posttest differentially affects the different comparison groups Selection-instrumentation effect May exist if the nature of the dependent variable or the way it is measured varies across the nonequivalent groups Selection item Test item or task to which the students respond by selecting their answers from choices given

Assessment Glossary


157

Selection-maturation effect Occurs when different comparison groups experience a different rate of change on a maturation variable Selection-regression effect May exist if the two groups are from different populations (e.g. as when the experimental treatment group is from a population of individuals with low reading scores and the comparison group from a population of individuals with high reading scores) Selection response A response style where the participant selects from a pull-down list Selective coding The final stage in grounded theory data analysis Selective reduction Central idea of content analysis; Text is reduced to categories consisting of a word, set of words or phrases, on which the researcher can focus; Specific words or patterns are indicative of the research question and determine levels of analysis and generalization Self adaptive Category of computer-adaptive tests in which the test taker selects to receive a more difficult or easier item based on feedback regarding their performance on the previously administered item Self-assessment An assessment method in which students make and report judgments about their own performance; The process of evaluating one's own learning; Often includes the ability to judge one's own achievements and performances, understanding how the product or performance was achieved, understanding why one followed the process one did, and understanding what might be done to improve the process, product, or performance; The process of judging one's own teaching performance for the purpose of self-improvement; Instructional self-assessment techniques include: self-viewing on a videotape, observing and modeling exemplary teachers, filling out self-rating forms, completing open-ended self-reports, keeping a log, compiling a portfolio, or using self-study handbooks and materials Self-concept A person's judgment of his or her competence or skills in comparison to those of others Self-completion or self-administered A questionnaire designed to be filled in by respondents; A survey using a questionnaire designed to be filled in by respondents Self-efficacy Students’ judgment of their own capabilities for a specific learning outcome Self-esteem Overall assessment of personal adequacy or worth Self-evaluation The process of reviewing one's own behavior and student learning outcomes for the purpose of monitoring and changing one's own teaching performance Self-report Student evaluations of their own learning or abilities; A test-taking method in which the participants check or rate the degree to which various characteristics are descriptive of themselves

Assessment Glossary


158

Self-report instrument or measures A device in which persons make and report judgments about the functioning of their project, program, or instructional material; Those instruments in which teachers record their own recollections of events, feelings, judgments, and attitudes Semi-structured interview An interview that uses a list of topics to be covered instead of a fixed questionnaire Semantic differential Rating scale which contains a series of items with each concept bounded by bipolar adjectives representing contrasting views of the performance or product; A scaling technique in which participants rate a series of objects or concepts Sensitivity A mathematically derived index of how good a test is at detecting true positives; Instruments with high sensitivity are called for when false negatives are either expensive or dangerous because they result in missed or delayed opportunities for improvement, but they may result in a high number of false positives; The awareness and understanding of other people's feelings, attitudes, social and cultural backgrounds, ethnic traditions and customs, languages, interests, rights, and needs Sequence response A response style where the participant orders a list of objects or text to formulate their response Sequential time People with this view regard time as incremental, do one thing at a time, make appointments and generally stick to schedules Sequencing effects Biasing effects that can occur when each participant must participate in each experimental treatment condition Serial effect In survey research, a situation where questions may "lead" participant responses through establishing a certain tone early in the questionnaire; The serial effect may accrue as several questions establish a pattern of response in the participant, biasing results Service delivery The method used to provide services to a client Service learning Typically student affairs program related to enhanced social responsibility through community based environments Setting The temporal and physical environment of an event or activity Shared beliefs The specific cultural conventions or statements that people who share a culture hold to be true or false Shared values The culturally defined standards about what is good or bad and desirable or undesirable Short-term debt

Assessment Glossary


159

Debts payable in full within a period of one year or less Short-term orientation The cultural perspective that values quick and immediate results, not long-term benefits Significance Refers to the likelihood that relationships observed in a sample could be attributed to sampling error alone Significance level The cutoff the researcher uses to decide when to reject the null hypothesis; Usually expressed as a p-value; Also called the alpha level Significance testing A commonly used synonym for hypothesis testing; In significance or hypothesis testing, the researcher sets the significance (alpha) level, analyzes the data to obtain the probability value, and then the researcher compares the empirical probability value with the preset significance level to determine whether the finding is statistically significant Significant difference A decision that an observed difference between two statistics probably did not occur by chance Similar-to-me effect A type of rater effect in which an assessor or an evaluator judges more favorably those people seen as similar to himself/herself Simple case When there is only one independent variable and one dependent variable Simple case of causal-comparative research When there is one categorical independent variable and one quantitative dependent variable Simple case of correlational research When there is one quantitative independent variable and one quantitative dependent variable Simple random sample A sample drawn by a procedure where every member of the population has an equal chance of being selected Simple random sampling The term usually used for random sampling without replacement Simple regression Regression based on one dependent variable and one independent variable Simulation An imitation of a typical task or situation to assess how well a learner/teacher might implement such a task or perform in an actual situation; Competency-based measure where a person's abilities are measured in a situation that approximates a "real world" setting; Primarily used when it is impractical to observe a person performing a task in a real world situation (e.g. on the job) Simulation study A study in which the probable effects of alternative solutions to a problem are identified by using symbolic representations of real activities, situations, or environments

Assessment Glossary


160

Single-case experimental designs Designs that use a single participant to investigate the effect of an experimental treatment condition Single Blind A study in which either the experimenter(s) or the participants are ignorant of the true purpose of the research; See also Double blind Single case study The researcher explores a single entity or phenomenon ("the case") bounded by time and activity (a program, event, process, institution, or social group) and collects detailed information by using a variety of data collection procedures during a sustained period of time Situational specificity The extent to which it is appropriate to use an assessment conducted in one setting or context for other settings or contexts Skewed Not symmetrical Skewed distribution Any distribution which is not a normal distribution; A distribution that is not symmetrical along the x-axis Skill or skills The ability to use knowledge in a practical manner; Observable behaviors that demonstrate levels of competence (i.e., knowledge, comprehension, application, analysis synthesis, and evaluation) SLO See Student learning outcome Small group A collection of three or more individuals who interact with one another to achieve a common task or goal SMART goals, objectives, or outcomes Specific, Measurable, Achievable, Relevant, and Time bound; Specific, Measurable, Attainable, Realistic, and Timely; Specific, Measurable, Agreed upon, Realistic, and Time based; Specific, Measurable, Attainable, Relevant, and Time sensitive; Specific, Measurable, Acceptable, Realistic, and Targeted; Specific, Measurable, Attainable, Results oriented, and Targeted; Specific, Measurable, Action oriented, Reasonable, and Time specific; Specific, Measurable, Actionable, Relevant, and Timed; Specific, Measurable, Aggressive but Attainable, Results oriented, and Time bound; Specific, Measurable, Attainable, Rewarding, and Timed; etc. Snowball sampling Each research participant is asked to identify other potential research participants Social awareness Students' sense of how they are related to human society as a whole; Implies sensitivity to the welfare of other members of our society; Students might demonstrate social awareness by, for example, joining a group to clean up the environment Social capital The part of the intangible capital of an organization that relates to personal networks, mentoring relationships, know-who and corporate culture Social desirability response set

Assessment Glossary


161

The tendency to provide answers that are socially desirable Sociodrama Dramatization and role playing used to teach an audience about the findings of an evaluation and to illustrate their potential applications Solomon four group design An experimental design using three control groups to avoid the possibility that the pre-test might itself affect the dependent variable (The traditional experimental group is given the pre-test, the main treatment, and the post-test in the normal way; the traditional control group is given the pre-test, a control treatment, and the post-test; a second control group is not pre-tested, but does get the main treatment and the post-test; and a third control group only gets the post-test); In this design, the differences between the experimental group and the second control group or those between the first and third control groups are assumed to be caused by the pretesting Solvency ratio One of the many ratios used to gauge a company’s ability to meet its long-term financial obligations based on current assets and liabilities Sourcing Information that identifies the source or attribution of the document Span of control One of the early principles of administration which states that there is an upper limit to the number of subordinates any administrator can directly supervise (generally set at twelve|) and advises administrators to eliminate any violations of this principle by reducing the number of officials reporting to them by either merging certain offices or stretching out the hierarchy Spearman-Brown formula A formula derived within classical test theory that projects the reliability of a shortened or lengthened test from the reliability of a test of specified length; A statistical formula used for correcting the split-half reliability coefficient (because of the shortened test length created by splitting the full length test into two equivalent halves) Specialist Defines an individual who has expertise and responsibility within a specific occupation, job function, field of study, or research Specific culture A culture that focuses on interactions for specific purposes and separates work, family, and social roles Specifications A delineation of the major attributes of an assessment to be developed, including breadth and depth of content to be covered, level of difficulty, format of the assessment materials, supplies and equipment needed, level of complexity, administrative process (e.g. individual or group, location, timing), scoring procedures, and numbers and types of items and tasks Specificity A measure of how good a test is at detecting negatives in a population of condition negatives; High specificity is called for in tests where false positives are either expensive or dangerous because false positives result in inappropriate treatments, unnecessary referrals, harder experiences, or failure Specimen set A sample set of testing materials that are available from a commercial test publisher; May include a complete individual test without multiple copies or a copy of the basic test and administration procedures

Assessment Glossary


162

Speed or speeded test A test in which performance is measured by the number of tasks performed in a given time; A test in which performance is measured primarily or exclusively by the time to perform a specified task; A test scored for accuracy where the test taker works under time pressure; A test is considered a pure speed test if everyone who reaches an item gets it right but no one has time to finish all items; Thus, score differences depend upon the number of items attempted Speededness A test characteristic, dictated by the test's time limits, which results in a test taker's score being dependent on the rate at which work is performed as well as the correctness of the responses; The term is not used to describe tests of speed; Speededness is often an undesirable characteristic Split-half reliability A measure of the consistency of the scores obtained from two equivalent halves of the same test; A statistical procedure by which a random half of the test items is correlated with the other half: Often a more useful measure than test-retest reliability because the data comes from a single test session and there will be no practice, illness progression, recovery, mood, or similar effects Split-halves reliability coefficient An internal consistency coefficient obtained by using half the items on the test to yield one score and the other half of the items to yield a second, independent score.; The correlation between the scores on these two half-tests, adjusted via the Spearman-Brown formula, provides an estimate of the alternate-form reliability of the total test (The Spearman-Brown formula is a formula derived within classical test theory that projects the reliability of a shortened or lengthened test from the reliability of a test of specified length); An estimate of reliability determined by applying the Spearman-Brown formula for m=2 to the correlation between two halves of the same test (e.g. the odd-numbered items and the even numbered items) Spontaneous performance assessment A measure based on observing, without prompting or pre-announcing, what a subject does during non-assessment activities; Measure based on accidental or unplanned observations which may not be representative or generalizable Spurious relationship When the relationship between two variables is due to one or more third variables Stability reliability The agreement of measuring instruments over time; The stability of an instrument is often expressed by the coefficient of stability which is used to express the degree of the relationship that exists between scores on the same test (or parallel forms of the same test) observed at two points in time Staffing metrics Measures used to determine costs associated with recruitment and hiring, time-to-fill and time-to-start for open positions, and recruiter workload/activity Stakeholders Any person legitimately involved in or affected by the evaluation; Individuals or groups who may affect or be affected by program evaluation; Those individuals who have a substantial interest in an assessment/evaluation and in the quality of learning/teaching; Anyone who has a vested interest in the outcome of a program, project, or plan; The various groups with an interest in the quality, governance, and operation of a program (e.g. the public, employers, customers, clients, and third party payers); Individuals and groups inside and outside the firm who have an interest in the actions and decisions of the firm; The many different persons or interest groups that are involved in a policy decision and are affected by the results; Those people, institutions and interests which

Assessment Glossary


163

have an interest in, or contribute to, the organization's success or failure in achieving its mission; See also Right-to-know audience Standard or standards A description of the expected level of performance that describes minimum competence in relation to a critical score or other measure of student performance; A principle commonly agreed to by experts in the conduct and use of evaluation for the measure of the value or quality of an evaluation; A pre-determined criterion or expectation of a level of student learning; A level of accomplishment all students are expected to meet or exceed (e.g. a passing score); Standards do not necessarily imply high quality learning, sometimes the level is a lowest common denominator; Standards do not imply complete standardization in a program since a common minimum level could be achieved by multiple pathways and demonstrated in various ways; There are several categories of standards related to different purposes: Desired performance standards (Specified levels of performance regarded as accomplished or effective and typically are used for such purposes as promotions, awards, and certification), Developmental standards (Specified improvement levels to be attained which may be used for professional development and self-assessment), Legal standards (Specified guidelines and requirements related to an assessment and evaluation that are specified in the law, governmental policies and regulations, school district policies, and court decisions), Minimum standards (Specified levels below which performance is not acceptable and are used for such purposes as licensure and job assignments), Professional standards (Specified guidelines related to an assessment and evaluation that are specified by the individuals and associations in the career area affected, directly or indirectly, by the assessment), and Technical standards (Specified guidelines related to an assessment and evaluation that are specified by psychometricians, statisticians, test publishers, and specialists in the domain covered by the assessment); See also Primary and Secondary standards Standard deviation (SD) A measure of the variability or dispersion of a distribution of scores; Most widely used measure of dispersion of a frequency distribution; The square root of the variance; The standard deviation is a calculated number that describes the extent to which scores are dispersed (spread out) from the mean; Nearly all scores are typically within 3 standard deviations of the mean; A measure of variation that indicates the typical distance between the scores of a distribution and the mean; Determined by taking the square root of the average of the squared deviations in a given distribution; Can be used to indicate the proportion of data within certain ranges of scale values when the distribution conforms closely to the normal curve; Equal to the positive square root of the population variance; The more the scores cluster around the mean, the smaller the standard deviation; For a normal distribution, approximately two thirds (68.3 percent) of the scores are within the range from one standard deviation below the mean to one standard deviation above the mean (often stated as plus or minus one standard deviation); Computation is based upon the square of the deviation of each score from the mean; Sometimes called sigma Standard error The standard deviation of a sampling distribution; Statistical estimate of possible size error present in a test score or other group measure Standard error of measurement The standard deviation of an individual's observed scores from repeated administrations of a test (or parallel forms of a test) under identical conditions; The standard deviation of the distribution of errors of measurement that is associated with the test scores for a specified group of participants; The difference between an observed score and the corresponding true score or proficiency; The standard deviation of an individual's observed scores from repeated administrations of a test or parallel forms of a test, under identical conditions; Because such data cannot generally be collected, the standard error of measurement is usually estimated from group data Standard error of the mean A computed value based on the size of the sample and the standard deviation of the distribution; Indicates the range within which the mean of the population is likely to be found from the mean of the sample at a given level of probability

Assessment Glossary


164

Standard operating procedures (SOP) A prescribed written procedure outlining how recurring tasks, duties, and functions are to be performed organization-wide Standard or standardized score A score that is expressed as a deviation from a population mean; A score expressing an individual's distance from the mean in terms of the standard deviation of the distribution; Generally requires the mathematic conversion of raw scores and a distribution that approximates a normal curve; Scores that have been converted from one scale to another to have a particular mean and standard deviation; An indicator of the relative standing of a score within a normal distribution of scores, defined by its mean and standard deviation; By transforming raw scores to standard scores, the user can interpret intervals between any two score points in relation to a reference population; Z scores are a commonly used standard score transformation, providing a normal distribution with a mean of zero and a standard deviation of one; If it is important to compare performance on different assessments, then the scores from each test or assessment need to be standardized since each assessment may have a different number of items or rating scale points and each may have been administered to different groups; Standardizing the scores within each distribution becomes necessary for purposes of comparing performance on different assessments; By transforming raw scores for each assessment into standard scores, it becomes possible to determine a learner/teacher's relative strengths and weaknesses within the groups upon which the scores were standardized Standard setting Process by which an expectation of performance level (standard) is determined Standardization sample That part of the population that is used in the norming of a test (i.e. the reference population); The sample should represent the population in essential characteristics, some of which may be geographical location, age, graduation, or participation in a specific type of program Standardized or standardization The use of consistent procedures for administering, scoring, reviewing, interpreting, and reporting the results of an evaluation; Presenting the same stimulus to all participants; In test administration, maintaining a constant testing environment, and conducting the test according to detailed rules and specifications, so that testing conditions are the same for all participants; In statistical analysis, transforming a variable so that its standard deviation is 1.0 for some specified population or sample; In scoring, ensuring that candidate responses are judged using predefined criteria in order to provide a consistent basis for evaluating all candidates; A consistent set of procedures for designing, administering, and scoring a measure; The purpose of standardization is to assure that all students are evaluated under the same conditions so that their scores or ratings have the same meaning and are not influenced by differing conditions; Standardized procedures are very important when scores and ratings will be used to compare individuals or groups; Design and implementation of consistent specifications for procedures, practices, materials, machinery or other equipment, or other types of products and services; The process of calibrating a measure to a given population so that the future performance of other samples can be norm-referenced Standardized conditions The administration of an assessment process or instrument to all learner/teachers in the identical manner (e.g. same instructions and timing, comparable settings, and use of trained assessors and evaluators) Standardized open-ended interview A set of open-ended questions are asked in a specific order and exactly as worded Standardized tests or testing A test composed of a systematic sampling of behavior, having data on reliability and validity, administered and scored according to specific instructions, and capable of being interpreted in terms of adequate norms; A test that is administered and scored under the same conditions for all students; A sample of items or situations with definite directions for administration and scoring most often accompanied by data on reliability and validity and

Assessment Glossary


165

sometimes by normative information; Assessment methods, either criterion- or norm-referenced, designed to be administered, scored, and interpreted in the same way regardless of when and where it is administered; An objective test that is given and scored in a uniform manner; Tests which are carefully constructed with items selected after trials for appropriateness and difficulty, issued with a manual giving complete guidelines for administration and scoring in an attempt to eliminate extraneous interference that might influence test results, and often with scores that are norm-referenced; A form of measurement that has been normed against a specific population; Standardization is obtained by administering the test to a given population and then calculating means, standard deviations, standardized scores, and percentiles, with equivalent scores produced for comparisons of an individual score to the norm group's performance Standards The broadest of a family of terms referring to statements of expectations for student learning, including content standards, performance standards, Opportunity to Learn (OTL), and benchmarks Standards-based reform Program of school improvement involving setting high standards for all students and a process for adapting instruction and assessment to make sure all students can achieve the standards Stanine or stanine score Abbreviation for “standard nine point scale”; A score from a nine-point scale used in standardized tests; Normalized standard scores with a range of 1 to 9, a mean of five, and a standard deviation of two; One of the steps in a nine-point scale of standard scores; They are status scores within a particular norm group where the first stanine is the lowest scoring group and the 9th stanine is the highest scoring group (generally the first stanine is classified as "poor", the second and third are grouped together as "below average", the fourth through sixth are classified as "average", the seventh and eighth as "above average", and the ninth as "superior”); The word stanine draws its name from the fact that it is a STAndard score on a scale of NINE units; In a normal distribution the nine stanines take up 4%, 7%, 12%, 17%, 20%,17%, 12%, 7%, and 4% respectively of the distribution Starting point A randomly selected number between one and k State An unstable cognitive, affective, or psychomotor characteristic possessed in different amounts by the same person at different times and situations; Distinguishable, but less enduring ways in which individuals vary Statement of financial position A financial statement of a company’s incomes and outflows during a certain period of time; A term generally used by nonprofit organizations as a reference to a balance sheet Statement of work (SOW) A detailed pragmatic statement of a company’s needs and requirements on which prospective suppliers base their bids or proposals to provide products or services Statistic A numerical characteristic of a sample; A quantity, property or fact that gives information about a sample; A summary number typically used to describe a characteristic of a sample and from which inferences about the population represented by the sample are made; A quantity calculated from a sample of data and used to give information about unknown values in the corresponding population; For example: the average of the data in a sample is used to infer information about the overall average in the population from which that sample was drawn Statistical analysis

Assessment Glossary


166

A mathematical process using statistical techniques to tabulate, depict and describe a collection of numeric (or quantitative) data Statistical conclusion validity The ability to infer that the independent and dependent variables are related and the strength of that relationship Statistical inference Using information from a sample to draw conclusions or inferences about the population from which the sample was taken Statistical significance A situation or occurrence where a researcher can show through specific tests for significance that it is unlikely that the obtained results occurred by chance Statistically significant The probability of a finding being wrong or random is nil (p-value of .05 or less); A research finding is probably not attributable to chance; The probability that the association between the factor and the outcome is due to chance is less than a specified level (usually p < 0.05, 0.01 or 0.001); The claim made when the evidence suggests an observed result was probably not just due to chance (i.e., there is a real relationship present) Status report A description of the implementation of the plan's assessment methods, the findings (evidence) from assessment methods, how the findings were used in decisions to maintain or improve student learning (academic programs) or unit outcomes (support units), the results of previous changes to improve outcomes, and the need for additional information and/or resources to implement an approved assessment plan or gather additional evidence Stem A question or statement followed by a number of choices or alternatives that answer or complete the question or statement; Most commonly found in multiple-choice questions Sticky dots A method of voting used in consensus groups and co-discovery conferences where each participant is given a fixed number of small round colored self-adhesive labels used to vote for statements on posters by putting a dot next to the statements they most agree with; Also called dotmocracy Stop out When a student leaves school for a term and then returns to school Straight line depreciation A depreciation method that produces a regular charge that is proportional to the life expectancy of a fixed asset Strands A way of organizing what adult learners need to know and be able to do within core curriculum; Strands need not be taught sequentially or separately; They identify processes, themes, content, and skills Strategic Maneuvering yourself into a favorable position to use your strengths to take advantage of opportunities Strategic alliances Two or more companies coming together to enhance business operations Strategic alternatives

Assessment Glossary


167

Alternative courses of action that achieve business goals and objectives by using organizational strengths to take advantage of environmental opportunities Strategic audit A checklist of questions that provide an assessment of a firm’s strategic position and performance Strategic decisions A series of decisions used to implement a strategy Strategic factors Strategic issues expected to have a high probability of occurrence and impact on the firm Strategic fit Fit between what the environment wants and what the firm has to offer Strategic goals Fundamental issues the university must address; They give direction for accomplishing the mission, contain specific and meaningful planning challenges, and result from previous analysis; Strategic goals may be SMAR T goals or desired ends which are not necessarily attainable or quantifiable Strategic issues Trends and forces which occur within the firm or the environment surrounding the firm Strategic management The act of identifying markets and assembling the resources needed to compete in these markets; The set of managerial decisions and actions that determine the long-run performance of the firm Strategic myopia Management’s failure to recognize the importance of changing external conditions because they are blinded by their shared, strongly held beliefs Strategic objectives Means of achieving, or moving toward, a goal that are measurable and quantifiable; They focus efforts on demonstrable results and provide broad categories for planning resource allocation; A results-oriented objective Strategic performance measure A yardstick or standard used to measure progress toward achieving a strategic objective; A measure of how well we are doing Strategic plan A practical, action-oriented guide based on an examination of internal and external factors that directs goal-setting and resource allocation to achieve meaningful results over time; A document used to organize the present on the basis of projections of the desired future Strategic planning The process of developing strategies to reach a defined objective; A comprehensive planning process designed to determine how the firm will achieve its mission, goals, and objectives over the next five, ten, or more years; Matching organizational objectives and capabilities to the anticipated demands of the environment to produce a plan of action that will ensure achievement of objectives; A continuous process where people make decisions about intended future outcomes, how outcomes are to be accomplished, and how success is measured and evaluated; The process of identifying an organization’s long-term goals and objectives and then determining the best approach for achieving those goals and objectives Strategic predisposition

Assessment Glossary


168

A tendency of a firm by virtue of its history, assets, or culture to favor one strategy over competitive possibilities Strategic priorities A ranking of an organization's strategic objectives by importance; A description of what an organization's decision-makers believe is more important or less important; A basis (criterion) for evaluating the tactical planning and budget-setting process Strategic staffing The practice of hiring smaller core numbers of permanent employees and using temporary employees to fill more highly specialized positions within the organization Strategic thinking Making decisions today which will affect organizational performance years in the future Strategy or strategies Broadly stated means of deploying resources to achieve strategic goals; A pattern in a stream of decisions and actions; Strategies may be dominant (optimal regardless of the action taken by one’s rival), emergent (an unplanned strategy that emerges from within the organization), intended (a planned strategy developed through the strategic planning process), and realized (the real strategy of a firm, whether planned or emergent) Stratification variable The variable on which the population is divided Stratified coefficient alpha A modification of coefficient alpha that renders it appropriate for a multi-factor test by defining the total score as the composite of scores on single-factor part-tests Stratified random sample A grouping achieved by dividing the population to be surveyed into a number of non overlapping classes or categories which together include all cases, followed by taking cases at random from within the categories; A set of random samples, each of a specified size, from several different sets, which are viewed as strata of the population Stratified sampling Dividing the population into mutually exclusive groups and then selecting a random sample from each group Stringency A type of rater effect in which an assessor or an evaluator tends to rate a learner/teacher too low or to judge the performance level as poorer than it actually is; Sometimes referred to as "negative leniency” Structural equation modeling (SEM) Like path analysis, a method for studying the direct and indirect effects of variables hypothesized as causes of variables treated as effects; Unlike path analysis, this method allows for causal pathways to be specified between factors in addition to manifest variables and it also allows for the estimation of measurement and latent errors; Structural equation modeling is thought to subsume path analysis as an analytic approach; See Causal modeling Structured interview The interviewer asks all respondents the same questions using an interview schedule Structured observation The researcher determines at the outset precisely what behaviors are to be observed and typically uses a standardized checklist to record the frequency with which those behaviors are observed over a specified time period

Assessment Glossary


169

Structured performance assessment A measure based on the administration of an assessment instrument or task at a specified time and in a designated manner Student affairs Division of a higher education institute charged with providing programs and services that assist students in growing, progressing, or increasing his or her developmental capabilities as a result of enrollment, usually in out-of-class activities Student competency Statement of broad knowledge, skills, attitudes, or behaviors that program majors should be able to demonstrate upon completion of the degree program Student development The ways that a student grows, progresses, or increases his or her developmental capabilities as a result of enrollment in an institution of higher education Student evaluation The process of systematically collecting and interpreting information that can be used to inform students about the progress they are making toward attaining the knowledge, skills, attitudes, and behaviors to be learned or acquired; and to inform the various personnel who make educational decisions (instructional, diagnostic, placement, promotion, graduation) about students Student evaluation system All the procedures (including developing and choosing methods for assessment, collecting assessment information, judging and scoring student performance, summarizing and interpreting results, and reporting evaluation findings) and the policies that evaluators use to evaluate their students Student involvement Student engagement in project, activities, or their own learning Student learning The acquisition of knowledge or behavior as a result of participation in programs and services Student learning outcome (SLO) Knowledge, skills, abilities, and attitudes that a student has attained at the end (or as a result) of his or her engagement in a particular set of collegiate experiences; What students are expected to produce by the end of a course, program, college experience, degree, or certificate program, or set of interactions with student services; Covers acquisition, retention, application, transfer, and adaptability of knowledge and skills; A general statement of the measurable skills and abilities a student will possess upon successfully finishing a course of study which is broader in scope than objectives and focuses on the important general abilities a successful student will take away from the class, program, or college for use in his/her future life; The measures of student achievement of knowledge, skills, and other educational outcomes such as improved student attitudes and behaviors that should have been taught to them by the teacher being evaluated Student load The sum of the award units of all courses in which a student is enrolled during a specified time period, as defined by the institution Student outcome The achieved results or the actual consequences of what a student has demonstrated or accomplished; May be academic or occupational accomplishments, as well as the intellectual, personal, civic development, attitudes, values, and beliefs that students attain as a result of postsecondary education

Assessment Glossary


170

Student outcomes assessment The act of assembling, analyzing, and using both quantitative and qualitative evidence of teaching and learning outcomes, in order to examine their congruence with stated purposes and educational objectives and to provide meaningful feedback that will stimulate self-renewal Student outcomes assessment cycle The process by which SLOs are identified, measured, and analyzed, and the results used to improve student learning is a three-step cycle: Identify: Instructors meet and work together in identifying what the students should produce at the end of a set of learning experiences (be it at the course or program level), SLOs are drafted, and an assessment plan written which states what the assessment will be (i.e. how the SLOs will be measured) and what the rubric will be (i.e. which standards will be used to determine attainment of the outcome); Assess: The semester following the identification of the SLOs and drafting the assessment plan, the proposed assessment is conducted and the data collected; and Reflect: In the third semester of the cycle, the data is compiled and the instructors come together again to discuss the results through dialogues which should include discussions about the meaning of the results and how they can be used to improve student learning. Student performance level (SPL) A standard description of an ESOL (English for Speakers of Other Languages) student's language ability at a given level in terms of speaking, listening, reading, writing, and the ability to communicate with a native speaker; A profile of skill levels for a student can thus be assigned and used for placement, instructional, or reporting purposes Student Satisfaction Inventory (SSI) A proprietary survey by Noel-Levitz that measures student satisfaction with a wide range of college services including academics, educational support, and administration; Frequently used as a companion to the Institutional Priorities Survey that measures satisfaction of faculty and staff in many of the same areas Students with disabilities (SWD) A broadly defined group of students with physical and/or mental impairments, such as blindness or learning disabilities, that might make it more difficult for them to do well on assessments without accommodations or adaptations Study Research study; The principle method for investigating the relationship among variables Style-based evaluation A teacher evaluation approach that uses criteria which specify in what way or how teachers are to perform their job (e.g. announced objectives at the start of each lesson, use of cooperative learning techniques) rather than what teachers are responsible for doing Subculture A culture embedded within a larger culture; A smaller grouping within a culture that has a set of similar characteristics which distinguishes it from the larger culture Subject An individual who is studied Subjective A characteristic of an assessment, observation, or conclusion that yields results which cannot be empirically verified by another person Subject matter expert (SME) An individual who has expertise in a business process or specific area

Assessment Glossary


171

Subjective measures Tests or measures that are scored or rated with some degree of subjective judgment (e.g. a performance rubric) Subjective scoring Different scorers and raters may differ on a student's score or rating; Most often associated with constructed response assessments Subjective test A test in which the impression or opinion of the assessor determines the score or evaluation of learner performance or achievement; A test in which the answers cannot be known or prescribed in advance Subsidiary A company that has more than half of its stock owned by another company or is completely owned by another company Succession planning The process of identifying long-range needs and cultivating a supply of internal talent to meet those future needs; Used to anticipate the future needs of the organization and assist in finding, assessing, and developing the human capital necessary to the strategy of the organization Sufficiency A judgment on whether an assessment task is comprehensive enough to produce a sample of student work broad enough in depth, relative to a body of knowledge or skill, to be considered an adequate measure of whether the student has attained the knowledge or achieved the skill; For forced choice assessments, the number of items used to decide this is the crucial issue for sufficiency Summary A concise report encompassing the major results of an assessment or evaluation Summary annual report A summarized report containing information on the financial status of an employee benefits plan Summative assessment, evaluation, or research Evaluation focused on determining the overall effectiveness and usefulness of the evaluation object; Evaluation at the conclusion of a unit or units of instruction or an activity or plan to determine or judge student skills and knowledge or effectiveness of a plan or activity; Assessment of outcomes which are the culmination of a teaching/learning process for a unit, subject, or program; An evaluation designed to present conclusions about the merit or worth of a student's performance; Accountability-oriented assessment; Evaluation designed to present conclusions about the merit or worth of an object and recommendations about whether it should be retained, altered, or eliminated; Culminating assessment for a unit, grade level, or course of study providing a status report on mastery or degree of proficiency according to identified learning outcomes; The use of data assembled at the end of a particular sequence of activities, to provide a macro view of teaching, learning, and institutional effectiveness; The gathering of information at the conclusion of a course, program, or undergraduate career to improve learning for the next cohort of students taking the course or program; Assessment or evaluation that is used to show learner achievement and is done at the conclusion of a course or some larger instructional period (e.g. at the end of the program); The purpose is to determine success or to what extent the program, project, and/or course met its goals; Formative and summative evaluations should be used to complement each other. Sunshine law Statutes granting the public access to government agency meetings and records Supply and demand

Assessment Glossary


172

The amount of goods or services available to consumers at a specific price and the level of consumer need or demand for those goods or services at the specified price Support unit review A process employed to examine the operational effectiveness (strengths and weaknesses) and student learning support of non-instructional units or multi-unit functions; Non-instructional equivalent of program review. Surrogate task A substitute or replacement for an actual activity or task, but one that is very similar to the actual task and that uses the same types of skills, behaviors, materials, and equipment Survey Research tool that includes at least one question which is either open-ended or close-ended and employs an oral or written method for asking these questions; The goal of a survey is to gain specific information about either a specific group or a representative sample of a particular group; Results are typically used to understand the attitudes, beliefs, or knowledge of a particular group Survey instrument The research tool used to conduct a survey (e.g. a mailed questionnaire, a telephone script, etc.) Survey research A term applied to non-experimental research based on questionnaires or interviews SWOT analysis An abbreviation used to denote analysis of an organization’s internal Strengths and Weaknesses and external Opportunities and Threats; Analysis of the strategic factors within an organization (Strengths and Weaknesses) and within the organization’s environment (Opportunities and Threats) usually summarized in a 2x2 matrix or table; Also called an Internal/external assessment Synchronic reliability The similarity of observations within the same time frame; NOT the similarity of things observed Synchronous time People with this view often see time as flexible and intangible with commitments desirable but not absolute Synthesis The selection, organization, and analysis of the materials collected; The process of combining data and information from multiple sources, or of ratings and judgments on separate scoring dimensions in order to arrive at a conclusion or result; Combining elements into a whole by using an original structure or solving a problem that requires a combination of several principles sequentially into a novel situation System Set of regularized interactions configured or "bounded" in a way that differentiates and separates them from other actions that constitute the system's environment Systematic Any methodical procedure or plan; In higher education, a systematic approach to management implies a careful analysis of choices about organization, effectiveness, priority, and decision-making in order to allocate scarce resources Systematic error A score component (often observed indirectly), not related to the test performance, that appears to be related to some salient variable or sub grouping of cases in an analysis; An error that is present every time an instrument is used

Assessment Glossary


173

Systematic innovation The purposeful and organized search for changes and the systematic analysis of the opportunities these changes might offer for economics and social innovation Systematic sample A sample obtained by determining the sampling interval, selecting a random starting point between 1 and k, and then selecting every kth element Systemic validity The negative and positive consequences of testing that need to be monitored in order to evaluate the long-range value of the test Systems appraisal Review of institutional learning over time (generally 3 years) Systems perspective Recognition that organizations are complex social systems with dynamically interrelated and interdependent groups of individuals, structuresm and processes that interact with each other and their environment; Holistic consideration of the contextual functioning of interacting elements within a system and the relations of the system to other systems in its environment Systems theory A theory of social organization, holding that organizations (like other organisms) behave according to inputs from their environment, outputs resulting from organizational activity, and feedback leading to further inputs; Also holds that changes in any one part of a group or organizational system affects all other parts; A framework by which one can analyze and/or describe any group of objects that work in concert to produce some result; Interdisciplinary study of the nature of complex systems in nature, society, and science

Assessment Glossary


174

T t-score A derived score on a scale having a mean score of 50 units and a standard deviation of 10 units t-test Statistical test used to determine if the scores of two groups differ on a single variable (e.g. to determine whether writing ability differs among students in two classrooms) t-test for correlation coefficients Statistical test used to determine whether a correlation coefficient is statistically significant t-test for independent samples Statistical test used to determine whether the difference between the means of two groups is statistically significant t-test for regression coefficients Statistical test used to determine whether a regression coefficient is statistically significant Table of random numbers A list of numbers that fall in a random order Table of specifications A two-way grid, sometimes called a test blueprint, that lists major areas of content to be covered by the test as row headings and major kinds of abilities to be developed and tested as column headings Tactical objective A written statement describing an intended output; A product-oriented or productivity-oriented objective; Describes how a strategic objective will be accomplished. Tactical performance measure A yardstick or standard used to measure progress toward achieving a tactical objective Tactical planning A continuous process where people make decisions about how outcomes are to be accomplished, what products will be produced, how success is measured and evaluated, and how budgetary resources are allocated Tactical priorities A ranking of an organization's tactical objectives by importance; Descriptions of which organizational outcomes its decision-makers believe are more or less important in achieving the organization's mission; A basis for determining the organization's operating budget Talent An unusually high level of proficiency in performing a task or using a skill; May be in the affective, cognitive, and/or psychomotor domains Talent development A genuine intent to foster the long-term learning or development of others; Includes the ability to judge talent (using a perspective that is detached from one's personal opinions) and capitalize on its potential, with the focus on the developmental intent, not one's ability or willingness to train

Assessment Glossary


175

Tangible net worth The sum total of a company’s assets less intangible assets and financial liabilities Target cash balance The amount of cash a company would like to have readily available Target population The larger population to whom the study results are to be generalized Task An activity, exercise, or question requiring students to solve a specific problem or demonstrate knowledge of specific topics or processes; A goal-directed assessment activity or project, which prescribes that the student use their background knowledge and skill in a somewhat long-term process to solve complex problems or answer a multi-faceted question; A complex activity requiring multiple responses to a challenging question or problem; A single question, problem, or activity used to assess a student; An assessment activity or assignment to be completed by the learner/teacher being evaluated Task analysis Defining standards and conditions of a particular task and identifying the distinguishing factors between tasks Task force Group brought together to work on specific organizational problems Task list The list of steps in a project Taxonomies In the context of SLOs, a taxonomy is a classification of skills and abilities based on level of achievement, where the levels move from very simple processes to very complex ones; Levels range from relatively simple recognition of value to complex processes such as critical evaluation of ethical situations; Taxonomies may be cognitive (classifications of cognitive skills and abilities ranging from relatively simple tasks such as recognition to very complex processes such as critical analysis), psychomotor (classifications of physical skills and abilities ranging from relatively simple imitation to creative mastery), or affective (classifications of skills and abilities that relate to values and emotions) Teachable The practicality or feasibility of providing instruction on a topic, knowledge area, or skill in terms of the readiness of the learners, the expertise of the teachers, and the availability of resources (e.g. equipment, time, funding) Teacher effectiveness Attribute of a teacher who has the capability or potential of having a positive impact on student learning, behavior, and attitudes Teacher evaluation Systematic appraisal of a teacher's performance and/or qualifications in relation to the teacher's defined professional role, job responsibilities, and school or district mission Teacher evaluation system A complete approach to the evaluation of teachers including its purpose, the rules and regulations that apply, the target group to be evaluated, the domains to be covered, the procedures and methods to be employed, the instruments to be used, the persons to be involved, and the types of reports and feedback to be provided Teacher improvement

Assessment Glossary


176

The accomplishment of goals and objectives for professional development, growth in knowledge, acquisition of skills, and changes in practices Teacher norms The expectations of teacher behavior in a given context or setting that are usually learned during student teaching or in the first year of teaching in a new school Teacher outcomes The impact of intervention or experience on teachers' knowledge, attitudes, behaviors, or skills Teacher-outcomes evaluation Approach to evaluating teachers based on student performance, usually with standardized achievement test results Teaching-improvement loop Teaching, learning, outcomes assessment, and improvement defined as elements of a feedback loop in which teaching influences learning, and the assessment of learning outcomes is used to improve teaching and learning Team teaching A system whereby two or more teachers pool their skills, knowledge, etc. to jointly develop, plan, and teach combined classes Telephone interview An interview conducted over the phone Temporal validity The extent to which the study results can be generalized across time Tenure Employment status conferred upon a teacher by state law or institutional regulation after successful completion of a probationary period which provides substantial, but not complete, protection against arbitrary or capricious dismissal and termination, and entitles the teacher to due process procedures and other protections that may not be available to the non-tenured teacher Tenured faculty Faculty who have completed a probationary period (usually two to five years) and are now considered permanent employees of an institution or district with all applicable benefits and rights as specified in state law, district policy, and collective bargaining agreements Tertiary sector The administrative and service sector of the economy; The part of the economy made up of nonprofit organizations Test An assessment instrument consisting of a sample of items or tasks from a particular domain and that can provide an estimate of performance in that domain; To administer an instrument or implement an assessment process; Set of questions or situations designed to elicit responses that permit an inference about what a student knows or can do; Tests generally utilize a paper and pencil format, occur within established time limits, restrict access to resources or reference materials, and yield a limited range of acceptable responses Test bias

Assessment Glossary


177

Used to refer to an instrument, which does not represent the ability or achievement of different groups equally; Biased instruments favor one group or groups over another group or groups even when controlled for ability level Test center A facility that provides computers and proctoring services in which to conduct tests Test documents Publications such as test manuals, technical manuals, user's guides, specimen sets, directions for test administrators and scorers, and previews for participants that provide the information necessary to evaluate the appropriateness and technical adequacy of a test for its intended purpose Test equivalence Assurance that examinees taking one version of a test do not have a relative advantage over those taking another version Test modification Changes made in the content, format, and/or administration procedure of a test in order to accommodate test takers who are unable to take the original test under standard test conditions Test-retest reliability A measure of the consistency of scores over time; The extent to which two administrations of the same test to the same group of subjects yield consistent results; Used to assess the consistency of a measure from one time to another; A reliability coefficient obtained by administering the same test a second time to the same group after a time interval and correlating the two sets of scores; The degree of agreement is the square of the correlation coefficient derived from correlating results from the same test administered twice to the same subjects, generally at least three months apart; Test-retest reliability coefficient A type of reliability coefficient obtained by administering the same test a second time, after a short interval, and correlating the two sets of scores; When the time interval between the two testings is considerable, such as several months, a test-retest reliability coefficient also reflects the stability of the examinee trait being measured in addition to the consistency of measurement provided by the test; Sometimes the term test-retest is also used to describe the administration of different forms of the same test, in which case this reliability coefficient becomes the same as the alternate form coefficient (see Parallel forms) Test score pollution An inflation of test scores that is the result of practices, usually associated with high-stakes testing, which are designed to increase test scores or performance ratings without improvement of actual performance on the attributes being assessed Test theory Includes the logic and statistical models that underlie standard practices in test use and design including methods devised to estimate, overcome, or minimize the extent to which extraneous problems or errors influence measurements taken in a given situation Testing Measurement of educational variables Testlet A group of items within a test that measure the same factor or which relate to the same stimulus; Items that are part of a testlet are not statistically independent Think tank

Assessment Glossary


178

A group organized for the purpose of intensive research and problem solving, especially in the areas of technology, social or political strategy, or demographics Theoretical sensitivity When a researcher is effective at thinking about what kinds of data need to be collected and what aspects of already collected data are the most important for the grounded theory Theoretical saturation Occurs when no new information or concepts are emerging from the data and the grounded theory has been validated Theoretical validity The degree to which a theoretical explanation fits the data Theory An explanation or an explanatory system that discusses how a phenomenon operates and why it operates as it does; A generalization or set of generalizations used systematically to explain some phenomenon; A particular interpretation of a body of empirically verified observations; An attempt to make sense of a number of confirmed hypotheses by drawing them together into a more meaningful whole Theory-based evaluation Faculty evaluation approach based on certain theories of teaching or of learning Theory triangulation The use of multiple theories and perspectives to help interpret and explain the data Theory x and theory y Two opposing assumptions about people at work that lead to opposing styles of management; Theory X assumes that most people hate work, avoid responsibility, prefer to be directed, and have to be controlled and coerced to put out a fair day's work; Theory Y assumes that people will seek responsibility, demonstrate a high degree of imagination, and exercise self-direction if they have a creative, challenging job to which they can become committed Thick description A rich and extensive set of details concerning methodology and context provided in a research report Think-aloud technique Participants verbalize their thoughts and perceptions while engaged in an activity Thinking skills Higher-order cognitive skills that enable human beings to comprehend experiences and information, apply knowledge, express complex concepts, make decisions, criticize and revise unsuitable constructs, and solve problems Third party Person(s) other than those directly involved in the educational process (e.g. employers, parents, consultants) Third variable A confounding extraneous variable Third variable problem An observed relationship between two variables that may be due to an extraneous variable

Assessment Glossary


179

Three necessary conditions for causation In the transference theory of causation, the three things that must be present to contend that causation has occurred are the relationship condition (variable A and variable B must be related), the temporal antecedence condition (proper time order must be established), and the lack of alternative explanation condition (the relationship between variables A and B must not be due to some third variable) Thrift institution The general term used to describe banks, lending institutions, and credit unions Time-interval sampling Checking for events during specific time intervals Time management The discipline of using time efficiently and well in order to achieve professional, personal, or organizational objectives Time series study or analysis A study in which periodic measurements are obtained prior to, during, and following the introduction of an intervention or treatment in order to reach conclusions about the effect of the intervention; A research design in which behavior of subjects in naturally occurring groups is measured periodically both before and after introduction of a treatment; Multiple repeated measures Time-to-degree The average number of semesters enrolled between student's first term and the term they graduate (elapsed semesters) Timed Tests A test administered to a test taker who is allotted a strictly prescribed amount of time to respond to the test Timeline A calendar or list of dates showing the evaluation stages and activities, and indicating the dates by which they should be implemented and be completed Timeliness Coming at an opportune time, or providing information at a point when it can readily inform the assessment/evaluation or decision-making process Tolerance Recognition of and respect for the opinions, practices, or behaviors of others Topology Mapping of the relationships among subjects Total debt A metric used to establish a company’s financial risk by determining the total amount of assets being financed by debt; Calculated by adding short-term and long-term debt and then dividing by total assets Total quality management (TQM) TQM is a management system taken from business and now being applied to higher education; TQM applies concepts of control, quality, process, and customer service to management; The quality movement brings with it a sense of collective responsibility for learning, a habit of listening to the people we serve, a preference for data, an ethic of continuous improvement, a determination to develop fully the talent of every learner, and an acknowledgement that we are professionally accountable to one another and to those we serve for results; A structured system that satisfies internal and external customers and suppliers by integrating the business

Assessment Glossary


180

environment, continuous improvement and breakthroughs with development, improvement, and maintenance cycles Track record A summary of past events and accomplishments related to an individual's performance Tracking A form of monitoring where a series of surveys which repeat a set of questions are administered over time to measure the change in responses Tracking systems Systems set up to monitor progress, compile management information, and keep goals on track Trade secret Any formula, pattern, device, or compilation of information used in one’s business that gives the individual or entity an opportunity to obtain an advantage over competitors that do not know or use this formula, pattern, device, or compilation of information Trademark The name of a particular product or service legally registered as the exclusive property of a specific business enterprise Traditional or conventional assessment Assessments utilizing traditional testing instruments; Traditional testing instruments include: Forced-choice, machine-scorable, pencil-and-paper tests (e.g. matching, true-false, multiple choice) and restricted-completion, short-answer, or essay questions Training The provision of instruction and planned activities to facilitate the learning of specific knowledge, skills, abilities, attitudes, and behaviors; Intervention involving the provision of information, materials, or opportunities for practice and experiment on specific aspects of teaching or learning Traits Distinguishable, relatively enduring ways in which one individual differs from another Transcript analysis Reviewing a student’s transcript file to answer questions about a student’s educational experiences and to help faculty examine and understand the patterns of courses that best enhance the learning and development of a student Transcription Transforming qualitative data into typed text Transferability Ability to apply the results of research in one context to another similar context; The extent to which a study invites readers to make connections between elements of the study and their own experiences; The degree to which the knowledge and skills demonstrated in solving an assessment task can be used in solving other work-related tasks and real-world activities; One of several characteristics used to evaluate assessments Transportability The appropriateness of extending the use of a policy, instrument, assessment procedure, or evaluation system across different teachers, student groups, subject areas, instructional approaches, learning activities, school settings, states, etc.

Assessment Glossary


181

Translation rules If one decides to generalize concepts during coding for a content analysis, then one must develop a set of rules by which less general concepts will be translated into more general ones Treatment The stimulus given to a dependent variable; Set of repeatable operations under the control of the experimenter that may be applied to experimental units or treatment groups; What researchers do to subjects in the experimental group, but not to those in the control group Treatment group In a study, the group of subjects who receive the intervention of interest Treatment variation validity The ability to generalize across variations of the treatment Trend A general direction or movement; In statistics, a trend is a statistically detectable change over time Trend analysis The process of forecasting an organization’s staffing needs by analyzing past employment patterns in order to identify trends that may be expected to continue; Forecasting technique that relies primarily on historical time series data to predict the future; Detailed examination of a company's financial ratios and cash flow for several accounting periods to determine changes in a borrower's financial position; Scientific study of changes in social patterns (e.g. changes in fashion, technology, and consumer behavior) Trend study Independent samples are taken from a population over time and the same questions are asked Trial Application of one factor combination on one experimental unit Triangulation or triangulate The use of a combination of assessment methods in a study; A process of combining methodologies to strengthen the reliability of a design approach; The use of multiple sources and methods to gather similar information (e.g. an assessment that incorporated surveys, interviews, and observations); The attempt to obtain more valid results by using multiple sources of data about one aspect of performance, multiple methods of collecting data, and/or multiple interpretations of the same data; The collection of data via multiple methods in order to determine if the results show a consistent outcome; The building of multiple sources of information or ideas to support a central finding or theme; Using a variety of different research approaches or methods with similar but different results to plot, define, or approximate the parameters of the area, field, or realm wherein lies the actual or true results; Multiple forms of evidence that point to the same conclusion True negative A correct diagnostic judgment that an entity does not fall within a target category True positive A correct diagnostic judgment that an entity does fall within a target category True score or value The unattainable ideal of a score which has no measurement error; A hypothetical score that represents an assessment result which is entirely free of error; Sometimes thought of as the average score of an infinite series of assessments with the same or exactly equivalent instruments, but with no practice effect or change in the person being assessed across the series of assessments

Assessment Glossary


182

Turnkey program A tailor-made prepackaged program, developed by an external contractor for an organization that is ready for immediate operation without a need for modifications Two-stage cluster sampling A set of clusters is randomly selected and then a random sample of elements is drawn from each of the clusters selected in stage one Two-tailed test A non-directional application of one of the two-group inferential statistics (i.e. the t-test, the Wilcoxon, or the Mann-Whitney) where the statistical procedure has to deal with separation of the two distributions down both of the asymptotes (or "tails" of the bell curve) Type I error Rejecting a true null hypothesis; Error where the null hypothesis is rejected when it is true, thus causing the hypothesis to be accepted when it should be rejected Type II error Failing to reject a false null hypothesis; Error where the null hypothesis is accepted when it is false, thus causing the hypothesis to be rejected when it should be accepted Type technique Manipulating the independent variable by varying the type of variable presented to the different comparison groups Typical case sampling Selecting what are believed to be average cases Typology A classification system that breaks something down into different types or kinds Typology theory The examination of individual differences in how people view and relate to the world

Assessment Glossary


183

U Underserved populations Groups identified as potential clients who are not receiving the full extent of services an organization provides, often because of a history of exclusion due to race or ethnicity Undervalued An asset that is available for purchase at a cost that is lower than the product’s actual value Unemployment insurance (UI) A statutory benefit designed to provide workers who have been laid off a weekly income during short periods of unemployment funded by state and federal taxes paid by employers Unique case orientation Perspective adopted by many researchers conducting qualitative observational studies; Researchers adopting this orientation remember every study is special and deserves in-depth attention; Especially necessary for doing cultural comparisons Unobtrusive measures Measures that do not require the researcher to intrude into the research context; Unobtrusive measurement can reduce the biases that result from the presence of the researcher or measurement instrument; Unobtrusive measures also reduce the degree of control the researcher has over the type of data collected; Three types of unobtrusive measurement are Indirect measures, Content analysis, and Secondary analysis Unintended consequences Any unplanned or unanticipated outcomes that occur as a result of implementing an assessment or evaluation; For example: The use of student test scores as part of teacher evaluation results in test score pollution practices and increased teaching-to-the-test activities and less reliance on the curriculum guide for planning instructional activities and lessons Unit cost The per-item cost Unit of analysis The least divisible element on which measures are taken and analyzed Univariate analysis Studying the distribution of cases of one variable only Universalism The belief that common evaluative standards, rules, and practices should apply to everyone regardless of circumstance Universe The domain of all possible items for a given construct; The domain of all possible test items for a given construct Upper limit The largest number on a confidence interval Unrealized capital gain

Assessment Glossary


184

An increase or decrease in the value of a security that is not genuine because the security has not actually been sold Unrestricted sampling The technical term used for sampling with replacement Unsecured debt or loan A debt that is not secured by any property (collateral) Unserved populations Groups identified as potential clients who are not receiving any of the services an organization provides, often because of a history of exclusion due to race or ethnicity Unstructured interview An interview done without using a questionnaire or list of topics; Normally used when respondents are asked to describe an important or recent event in their life; The researcher asks open-ended questions which allows respondents to talk freely and to influence the direction of the interview since there is no predetermined plan about the specific information to be gathered from those being interviewed Unstructured observation The researcher uses direct observation to record behaviors as they occur, with no preconceived ideas of what will be seen; There is no predetermined plan about what will be observed Usability testing Methods that evaluate products by observing and measuring the ease and ability of the intended users of a product to achieve or accomplish the intended purpose of the product; Similar to the machine-user interface or man-machine research of ergonomics and human factors engineering; Currently most associated with the evaluation of websites, web applications, and computer interfaces or programs Uses and gratifications Form of audience research which studies how people use the experience or consumption of media programs (for information, aesthetic feelings, personal needs, social needs, and/or escapism); See also Reception analysis Utilitarianism An ethical approach that says judgments of the ethics of a study depend on the consequences the study has for the research participants and the benefits that may arise from the study Utility The extent to which an evaluation will serve the relevant information needs of students, educators, and other appropriate users; The extent to which an evaluation produces and disseminates reports that inform relevant audiences and have beneficial impact on outcomes; The practical value of an assessment/evaluation system with respect to such factors as time requirements, logistics, resources needed, costs and benefits, and applicable regulatory and institutional policies, as well as the technical concerns of validity and reliability; A characteristic of scoring criteria that ensures the criteria are diagnostic and can communicate information about performance quality with clear implications for improvement; An evaluation, often in cost-benefit form, of the relative value of using a test vs. not using it, of using a test in one manner vs. another, or of using one test vs. another test

Assessment Glossary


185

V Vagueness Uncertainty in the meaning of words or phrases Validation The process of gathering evidence that supports inferences made on the basis of test scores; The process of determining the appropriateness, meaningfulness, and usefulness of a measure, an instrument, or an assessment process, and of the inferences resulting from it; The process of investigation by which the validity of the proposed interpretation of test scores is evaluated Validity The accuracy of the inferences, interpretations, or actions made on the basis of test scores; Refers to whether or not an assessment measures what it is intended to measure; Extent to which test scores actually reflect what they were meant to measure; The capacity of a measuring instrument to predict what it was designed to predict; Stated most often in terms of the correlation between values on the instrument and measures of performance on some criterion; The degree to which inferences drawn about a student's knowledge, skills, attitudes, and behaviors resulting from the assessment methods used are correct, trustworthy, and appropriate for making decisions about students; The extent to which the test scores or responses measure the attribute(s) that they were designed to measure The soundness of the use and interpretation of a measure; Degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure; A method can be reliable, consistently measuring the same thing, but not valid; Validity has three components: Relevance (the option measures your educational objective as directly as possible), Accuracy (the option measures your educational objective as precisely as possible), and Utility (the option provides formative and summative results with clear implications for educational program evaluation and improvement): There are also several different types of validity: See also Concurrent validity, Consequential validity, Construct validity, Content validity, Criterion-related validity, Curricular validity, Evidential basis of validity, Face validity, Instructional validity, Predictive validity, Systemic validity Validity coefficient A measure of the degree of validity, usually expressed as the correlation between the measure in question and another measure or a variable; A correlation coefficient computed to provide validity evidence, such as the correlation between test scores and criterion scores Validity evidence Empirical evidence and theoretical rationales that support the inferences or interpretations made from test scores Validity generalization study An investigation of the degree of generalizability or of transportability of a policy, instrument, process, procedure, or evaluation system Value added The effects educational providers have had on students during their programs of study; The impact of participating in higher education on student learning and development above that which would have occurred through natural maturation, usually measured as longitudinal change or difference between pretest and posttest; A comparison of the knowledge, skills, and developmental traits that students bring to the educational process with the knowledge, skills and developmental traits they demonstrate upon completion of the educational process; The change in an attribute or product that can be linked to an intervention (e.g. the change in student test scores before and after completion of a course or the change in the effectiveness of a teacher's classroom management skills as a result of participation in a workshop series); Increase in learning that occurs during a course, program, or undergraduate education; May focus on the individual student (e.g. how much better a

Assessment Glossary


186

student can write at the end than at the beginning) or a cohort of students (e.g. whether senior papers demonstrate more sophisticated writing skills, in the aggregate, than freshmen papers); Requires a baseline measurement for comparison Value chain The process of manufacturing a product in its entirety, from the procurement of raw materials to the time of utilization Value dimensions Meaningful differences in key values and beliefs among cultures identified by empirical research Value proposition The core benefit of a product or service Value standards Statements that describe attitudes teachers would like students to develop towards learning Values clarification techniques Procedures used to help groups recognize the different values held in the groups, to discern conflicts among these values, and to consider how these conflicts might be resolved Values statement A narrative description that outlines an organization’s beliefs and guiding principles; A written statement describing the direction that indicates success or improvement in internal organizational characteristics, or in the expression of beliefs the organization holds; A document outlining and representing the core priorities in the organization’s culture Variable A condition or characteristic that can take on different values or categories; Any quantity that can assume more than one state or numerical value; Observable characteristics that vary among individuals; A quantity you can measure or question you can ask about each member of a population (The answer to the question for a particular individual is called the value of the variable for the individual, or simply an observation); A measured characteristic that can assume various values or levels; See also Discrete variable and Continuous variable Variable costs Total costs minus any fixed costs Variability Degree to which characteristics are observed to vary Variance A measure of variation within a distribution determined by averaging the squared deviations from the mean of a distribution; A measure of variability; The average squared deviation from the mean; A measure of the average deviation from the mean in squared units; The square of the standard deviation; A measure of variability of test scores, computed as the sum of the squares of the deviations of raw scores from the arithmetic mean, divided by one less than the number of scores; The smaller the variance, the closer the individual scores are to the mean Variance forecast A measure that utilizes a demand and availability forecast to determine whether an organization is able to meet future manpower needs Variate The quantitative measure of a variable

Assessment Glossary


187

Variation The dispersion of data points around the mean of a distribution Vertical management A traditional organizational structure consisting of primary functions (e.g. engineering, manufacturing, finance, etc.) with each function having its own manager Vertical organization An organizational structure consisting of many layers of management or other positions of authority Verification The process of checking the accuracy of data and information about someone's experience, training, performance, and other attributes Verisimilitude Having the semblance of truth; Probability that the research findings are consistent with occurrences in the "real world" Vested rights An individual’s right to pension benefits that are paid up and therefore are not contingent upon the employee’s continued service to the employer Vision statement A narrative description of a future state to help keep project teams focused; A concise statement of what needs to change, typically the promoter, sponsor, or leader’s agenda; A compelling, conceptual image of the desired future; A realistic, credible, attractive future for the institution that sets forth the ideal state the institution aims to achieve and provides long-term direction, guidance, and inspiration for the organization; A vision statement is a description of what an organization wants to become or hopes to accomplish in the future (typically in the next 10 years) Visioning See envisioning Visitor or event survey A survey of visitors at a venue or event Vocational and technical education Organized educational activities that offer a sequence of courses that provides individuals with the academic and technical knowledge and skills the individuals need to prepare for further education and for careers in current or emerging employment sectors; Includes a competency-based applied learning that contributes to the academic knowledge, higher-order reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and occupation-specific skills Volunteerism Organizational support, often in the form of paid leave or sponsorship, for employees pursuing volunteer opportunities or performing community services Voting participation A behavioral outcome associated with citizenship skills

Assessment Glossary


188

W WASC Western Association of Schools and Colleges Web based assessments or tests Assessments delivered via the Internet, or an Intranet, in which the items reside on a server and are packaged with HTML to allow a participant to respond using a browser; Used to refer to instruments (test, surveys, and questionnaires) which are deliverable over the World Wide Web and viewable through web browsers Web surveys Participants read and complete a survey form that is developed for and located on the web Weighted average An average of quantities that takes into account their relative importance in a data set instead of treating each quantity equivalently Weighted score or scoring A score adjusted by such factors as the importance of the attribute assessed to a performance, the reliability and validity of the assessment from which the score was derived, or a combination of such factors; A method of scoring a test in which the number of points awarded for a correct (or diagnostically relevant) response is not the same for all items in the test; The scoring formula may award more points for one response to an item than for another Weighting The amount of emphasis given to a particular set of information; For grading purposes weighting usually entails multiplying all scores for one component by a numerical value to increase the emphasis it receives over other data; A method to combine the results of two or more assessments used in calculating the percent who meet the standard for a learning outcome; If some assessments are deemed more important than the cut-scores on those assessments may be given greater consideration or weight in determining the overall performance level Wellness program Programs such as on-site or subsidized fitness centers, health screenings, smoking cessation, weight reduction/management, health awareness, and education that target keeping employees healthy, thereby lowering costs associated with absenteeism, lost productivity, and increased health insurance claims Wholly-owned subsidiary A company that is entirely owned by another company Windfall profit An unforeseen profit that occurs as a consequence of events beyond the recipient’s control Withholding tax Income tax withheld from an employee’s wages and paid directly to federal, state and local government agencies by the employer Within-stage mixed model research Quantitative and qualitative approaches are mixed within one or more of the stages of research Word attack skills Means by which a person recognizes and perceives the meaning of words

Assessment Glossary


189

Word response A response style where the participant enters a word to indicate their choice Work-based learning Supervised student learning that takes place at a work site through unpaid or paid activities Work ethic or behavior A set of personal attributes required for success at work; Includes the ability to identify established rules, regulations, and policies, to practice cost effectiveness and time management, to assume responsibility for decisions and actions, to display initiative and assertiveness, and to demonstrate a willingness to learn Work-readiness Individual’s preparation and awareness of the world of work, labor market, occupational information, values clarification and personal understanding, career planning and decision making, and job search techniques, including daily living skills, personal skills, work ethic and initiative, reliability, and accepting responsibilities involved in maintaining a job Work sample An assessment method that uses actual and typical on-the-job activities or tasks Work simulation A surrogate or imitation of a work sample task Work-to-education The process of acquiring greater learning and development to meet increased workplace need and demands Workforce flexibility The contribution of postsecondary education to the adaptability of the workforce to keep pace with change Workforce planning The assessment of current workforce content and composition issues used to determine what actions must be taken to respond to future needs Working capital The amount of cash available to a business for routine operations Written surveys or questionnaires Documents which ask individuals to share their perceptions about the study target, such as their own or others skills/attitudes/behavior, or program/course qualities and attributes. Writing skills Communicating ideas and information through documents such as letters, manuals, reports, and graphs

Assessment Glossary


190

X x Unknown quantity to be determined by completion of a mathematical formula X-axis Width; The horizontal axis of a two-dimensional Cartesian coordinate system; The horizontal axis on a graph, chart, or plot

Assessment Glossary


191

Y Y-axis Height; The vertical axis of a two-dimensional Cartesian coordinate system; The vertical axis on a graph, chart, or plot Y-intercept The point where the regression line crosses the Y-axis Year-end closing Financial reports prepared and released at the end of a company’s fiscal year

Assessment Glossary


192

Z Z-score A type of standard score scale in which the mean equals zero and the standard deviation equals one unit for the group used in defining the scale Z-axis Depth; The third dimension in a three dimensional plane or coordinate system Zero-balance account A type of bank account whereby funds are automatically transferred into it from another account when claims against it arise, versus maintaining funds in the account on a regular basis Zero-base budgeting Budget format that presents information about the efficiency and effectiveness of existing programs and highlights possibilities for eliminating or reducing programs by assuming that the minimum funding level for the agency is zero, thereby requiring agency administrators to justify all expenditures by the same standard of review that normally are applied only to new programs or increments above the base; A budgeting system that starts with no authorized funds as a starting point; In a zero-based budget, each activity or program to be funded must be justified every time a new budget is prepared and resources are allocated accordingly Zero growth An economic term indicating a decline in output for two or more consecutive quarters ZMET Zaltman Metaphor Elicitation Technique; An elaborate way of discovering people's feelings about a concept or issue by producing a picture of their mental associations

Date post:	16-Nov-2014
Category:	Documents
Upload:	todd-titterud
View:	521 times
Download:	0 times