The Portals (Coolfont) Workshop decided that:

• DLESE will be a two-level collection: – The unreviewed collection: a broad collection of

content which is relevant to Earth System education and meets minimum quality and technical standards

– The reviewed collection: a subset of high quality teaching and learning materials which have been rigorously evaluated

• The rationale for having a reviewed collection: – For the user: guaranteed high-quality resources, even

for a teacher without expertise in the field or time to “comparison shop”

– For the creator: inclusion in the reviewed section of DLESE can become a recognized stamp of professional approval

• The rationale for having an unreviewed collection:– For the user: access to a wider range of teaching and

learning resources

– For the library builders: a pool from which to select the reviewed collection

OK, so how do we decide what goes into the reviewed collection?

From the Portals (Coolfont) workshop:

• Selection Criteria:– Accuracy, as evaluated by

scientists

– Importance/significance

– Pedagogical Effectiveness

– Well-documented

– Ease of use for students and faculty

– Motivational/inspirational for students

– Robustness/Sustainability

Familiar review procedures:

• “Traditional peer review”

• “Traditional educational evaluation”

Traditional “Peer-Review”

• Reviewers are selected for their expertise by an editor.

• Reviewers examine the material, or a description of the material, in their home or office.

• Typically two

reviews.

What’s wrong with this picture?

Traditional “Peer-Review”

There are no students in this picture!

Traditional “Peer-Review”

“Traditional Educational Evaluation”

• Evaluator (reviewer) is selected by the developer.

• Evaluator observes students in another teacher’s classroom and/or administers evaluation instruments

• Typically one evaluator, several

classes of students.

What’s wrong with this picture?

“Traditional Educational Evaluation”

Evaluation by independent professionalevaluators is labor-intensive and expensive!

“Traditional Educational Evaluation”

Community Review ConceptPremises

• The materials in the “inner-circle” of reviewed, DLESE-stamp-of-approval-bearing resources must be classroom-tested.

– However, testimony from the creator of a resource that learning has occurred in his or her classroom is insufficient.

– It is not realistic to pay for professional evaluators to go into classrooms to evaluate whether student learning has occurred for

every potential DLESE resource. – Experienced educators can tell whether or not their own students

are learning effectively from an educational resource. – It is easier to answer: “Did your students learn?” than “Do you

think students would learn?”

Community Review ConceptPremises (cont’d)

• In order to be useful, DLESE has to contain lots of resources. Therefore it must grow fast.

• In the DLESE ecosystem, teachers, classrooms and students will be abundant resources.

• The rate-limiting resources in DLESE’s growth will be money, and the time of paid librarians/editors/gatekeepers and computer wizards.

• This is a digital library; we can and should take advantage of automated information gathering

Community Review ConceptProcedure

Coolfont Selection Criteria How to implement

Well documented Review by library staff

Importance/Significance More than N (threshold number tobe chosen) educators from DLESEcommunity tried this resource intheir classroom.

Pedagogical Effectiveness

Ease of use for students andfaculty

Inspirational or motivational forstudents

On-line questionnaire filled out byeducators who used resource intheir classroom

Accuracy, as evaluated byscientists

Invited review by a scientist,recruited by an editor

Robustness/sustainability Testing by a quality assuranceprofessional

Community Review ConceptProcedures (cont’d)

• What happens to the questionnaire information? – Creator receives all feedback from “YES” and “NO” respondents.

– Builders of Discovery System receive feedback from “NO” respondents.

– Suggestions typed in the teaching tips field are added to the Teachers’ section of the resource.

– “Editor” or “gate-keeper” is automatically notified and receives full packet of reviews when number of complete reviews exceeds N and the average, or weighted average, of the numerical scores exceeds Y.

Community Review ConceptProcedure

Coolfont Selection Criteria How to implement

Well documented Review by library staff

Importance/Significance More than N (threshold number tobe chosen) educators from DLESEcommunity tried this resource intheir classroom.

Pedagogical Effectiveness

Ease of use for students andfaculty

Inspirational or motivational forstudents

On-line questionnaire filled out byeducators who used resource intheir classroom

Accuracy, as evaluated byscientists

Invited review by a scientist,recruited by an editor

Robustness/sustainability Testing by a quality assuranceprofessional

Community Review ConceptStrengths

• Inclusive: The community builds the library.

• Scalable: Hundreds or thousands of resources can be classroom-tested.

• Thorough: All seven Coolfont/Portals selection criteria are applied.

• Economical: Scarce talents are applied at the end of the process, to the smallest number of items.

Community Review ConceptIssues

• How do we get educators to send in their reviews?

• How do we ensure that reviews come from bona fide educators?

• Would creators “spin” the review process by soliciting reviews from

their friends? • Would the merely-good early arrival tend to keep out the truly

excellent later arrival?

• Some topics are inherently less inspirational/motivational than others; how do we avoid filtering out resources on such topics?

• What about off-the-wall, or erroneous, or malicious reviews?

Traditional “Peer-Review” Traditional “EducationalEvaluation

Proposed “CommunityReview”

Reviewers are selected bytheir expertise by an editor.

Evaluator (reviewer) isselected by the developer.

Reviewers step forward fromthe community.

Reviewers examine thematerial, or a description ofthe material, in their homeor office.

Evaluator observes studentsin another teacher’sclassroom and/or administersevaluation instruments

Reviewers test the materialin their own classroom withtheir own students.

Typically two reviews. Typically one evaluator,several classes of students.

A large number ofreviewers/students ispossible.

Moderate cost High cost Moderate cost

Editor is supposed to filterout unfair or maliciousreviews.

Evaluator is a neutral party,neither student’s teacher norresource creator.

Unfair or malicious reviewscan be swamped out bynumerical weight of goodreviews.

How can I become part of DLESE?

• … as a resource creator/contributor

• … as a user

• … as a reviewer/tester

Continue the conversation at:

collections@dlese.org

or

http://www.ldeo.columbia.edu/dlese/collections