PAMA

Performance Assessment of Metacognitive Awareness

 by Marné B. Isakson, Ph.D.Kenneth Plummer, Ph.D.

 

Thursday, November 7, 2013 10:15-11:15 a.m.

White Hill Room, 4th FloorBoston Park Plaza Hotel

PAMA Presentation 

1. Introduction—the Need2. An Experience & Discussion3. The Model 4. The PAMA5. Reactions to the PAMA6. Psychometrics Properties7. Where from here?

Please Read the Paragraph on the Handout Provided

Read it to understand well enough for a class discussion and an essay test.

Feel free to mark it anyway you would like.

Leave the bottom blank.

“No, when I go to sea, I go as a simple sailor, right

before the mast, plumb down into the forecastle,

aloft there to the royal masthead. True, they rather

order me about, and make me jump from spar to

spar, like a grasshopper in a May meadow. And at

first, this sort of thing is unpleasant enough. It

touches one’s sense of honour, particularly if you

come of an old established family in the land, the

Van Rensselaers, or Randolphs, or Hardicanutes.

And more than all, if just previous to putting your

hand into the tar-pot, you have been lording it as a

country schoolmaster, making the tallest boys stand

in awe of you. The transition is a keen one, I

assure , from a schoolmaster to a sailor, and

requires a strong decoction of Seneca and the Stoics

to enable you to grin and bear it. But even this

wears off in time” (p. 3).

A Taste of PAMA

1. Create a basic list what you did to make sense?2. Select one thing you did from the list above 

that was particularly helpful in making sense of all or part of the text.

3. Describe this one thing you did –how you did that for this text.

4. Why was that a good thing to do for this text?  What id that action do for your understanding of this text?

2 min. Share with a Partner—2 to 3

If you had more time, interest, and a good reason for doing so, what else could you do to be sure you thoroughly understand this segment of text?

Can you identify the author?

What is metacognitive awareness?  

Why is metacognitive awareness important for reading difficult academic texts?  Readers can only learn from these texts by

recognizing breakdowns in understanding,  knowing what to do about it, and  consciously manipulating their approaches to make understanding happen. 

It is watching yourself thinking.Having a conversation with yourself—exploring, questioning, correcting in 

process, “and pursuing the dynamic power of your own mind” (Bain, 2012, p. 24).

It is planning, monitoring, regulating, and evaluating your thinking as you read to make sure you meet your purposes for reading.

How might your students have typically approached this task?  

How could you assess their metacognitive awareness?   

2 min. Share with a Partner—2 to 3

Knowing Mj

Metacognitive Ability to MjSuccessfully Read Mj

College Texts Mj

Applying Mj

Declarative Mj Procedural Mj Conditional Mj

What are the Mjstrategies? Mj

Why are the Mjstrategies Mjused? Mj

What are the Mjspecific steps Mjof each Mjstrategy? Mj

What are the Mjconditions that Mjdictate the use Mjof each Mjstrategy? Mj

*What is my purpose for reading this text? Mj*How important is this text in achieving my purpose? Mj*How much time do I have? Mj*What is my energy level for tackling this text in this way? Mj*How unfamiliar are the words/concepts in the text? Mj*At what point should I stop reading to deal with a comprehension Mjproblem? Mj

Executing Mj Implementing Mj

Actually Mjexecute the Mjstrategy step Mjby step in a Mjguided reading Mjsituation Mj

Adapt one's declarative, Mjprocedural, and Mjconditional knowledge of a Mjstrategy and to bring to Mjbear one's prior Mjexperiences with it to Mjachieve one's purposes for Mjreading an unfamiliar text. Mj

PAMA Mj

Creating the PAMA

 Conception – 2006-2010Conceived and designed the PAMA, created materialsPilot #1—2010, winter semester, 20 Ss. 

Ken & I partner for this research—2011, fall semesterModel developed and refinedPilot #2—fall semester, qualitative data, 54 Ss

 1st data set—2012, winter semesterMaterials revisedPilot #3, 60 SsRater training, protocol scoring

2nd data set—2013, winter semesterMaterials revised again, Pilot #4, 80 SsRater training, protocol scoring

Current Use—2013, fall semesterA mini-PAMA used as midterm exam

PAMAPerformance Assessment of Metacognitive Awareness

1. The instructions2. The protocols3. The rubric4. Student reactions5. Professor reactions

The InstructionsPerformance Assessment of Metacognitive Thinking

when Reading a Difficult Text

Purpose of this Test: To see the metacognitive decisions you make to read a challenging, informational text, by. . . 

• reading the text mindfully with evidence in the margins and on the text that you are deeply engaged in processing the text and integrating multiple strategies to do so.

• giving evidence at stopping points that you have processed the text for basic and deep understanding, including critical and creative thinking, and can explain why you are doing what you are doing.

 

 

• Time Frame:  one hour of the actual test during the regularly scheduled final in the computer lab.  Setting up will add to the time slightly.

• Scoring: We will be looking for evidence of metacognitive awareness, layered 

reading, deep comprehension, critical thinking, creative thinking, and mindful reading and coding.   

• Materials:

Cover Sheet with instructions (I provide)

Course handbook  (you bring)

Pencil (you bring)

A blank sheet of paper if needed for extra note-making and 

mental processing  (I provide)

A text (I provide)

Instructions:  You have learned many principles, strategies, and key concepts for reading academic texts well.  Today you demonstrate what you have learned.  You are to lift yourself to the highest level of reading ability to learn from an unfamiliar text using the strategies that make sense to you.  Here are the steps:

1. Open the form e-mailed to you: PAMA Final W13.  When you open it, it should say “Performance Assessment of Metacognitive Thinking when Reading a Difficult Text.”  

2. When you receive a hard copy of the text, read it showing full engagement by marking the text and doing what you would do to process a difficult information text.  Pencil in hand, capture your thinking on the text, in its margins, and on the extra paper in any way that makes sense to you.  Use any strategies that work for you to do the following:  Demonstrate basic understanding.  Demonstrate deep, scholarly thinking, Demonstrate interesting critical and/or creative thinking. 

3. When directed (four times), return to the form e-mailed to you and go to the number of the row for that “metacognitive break.”   You will have exactly 15 minutes per round. The first 5 minutes you must be reading and “co-authoring” the text as evidenced by your markings.  The last 5 minutes you must be typing your metacognitive thinking.  The middle 5 minutes is for wiggle room.  Most start typing immediately but you may read a bit more if you feel compelled to do so.  But don’t linger—move to typing as soon as possible.  

4.      Type your answers for each column on the form.  You will do the following:

Step 1: List all the strategies you were aware of using since the last interruption. 

Step 2: Select one of these strategies to tell about in more detail.  Across the test, show a variety of strategies that have been particularly useful to you in processing and understanding this text with these criteria:

Do not pick the same strategy more than once (though you may use it and list it more than once; just don’t pick it more than once to describe as part of Steps 3-4).  

Across the four sessions, select two strategies that generate deep thinking—select the scholarly strategies learned since midterm as well as My Questions and Be the Teacher learned before midterm.  

Across the four sessions, choose a strategy that represents one of the three layers of reading difficult texts so that each layer is represented at least once.

Step 3: Describe how you used this strategy WITH THIS TEXT.  Describe in detail.  (HOW, UNDER WHAT CONDITIONS, HOW MODIFIED FOR THIS TEXT, etc.) 

 Step 4: Give the reason(s) why you used this strategy for processing THIS PART OF THE TEXT.  I want to see your metacognitive thinking.  (WHY, EVALUATE PROCESSING, WHY IT IS WORKING OR NOT WORKING FOR THIS TEXT, etc.)

 Save your work along the way.

 

5.      Continue reading the text.           You will have 60 minutes to process the text and to take the              Metacognitive Breaks.  Use every minute.

6. Since you have so long to process and learn from this short text, you may come to the point where you have  learned the meanings in the text as well as you feel necessary but you still have more time.  Move to critical reading and to creative reading and continue processing the text in these advanced ways.

7. When time is called, SAVE your work one last time and submit it as         instructed.

The Protocols

Step 1 Step 2 Step 3 Step 4

Metacog-nitive 

Thinking Session

List strategies you have used since the last 

interruption

Select ONE strategy you listed in Step 1 that was especially helpful in understanding this portion of the text.  What layer of reading is it?

Describe how you used this strategy WITH THIS TEXT.

Give the reason(s) why you chose this strategy for processing THIS PART OF THE TEXT. 

5 min to read and process 10 min to complete the four steps - >

1 

       

5 min to read and process 10 min to complete the four steps - >

2

       

5 min to read and process 10 min to complete the four steps - >

 3

       

5 min to read and process 10 min to complete the four steps - >

4

       

Name______________________________________Date________ Section ____________Text_____________________

Performance Assessment of Metacognitive Thinking when Reading a Difficult Text

[1] When considering the four strategies to discuss across the five rows:•TWO of your choices must generate higher-level thinking (scholarly strategies); that is, twice choose strategies that you learned since the midterm. So, after you list the strategies •you used in Step 1, select one that demands more thinking than the others for two of the four rows. The other two rows may be the less demanding strategies (survival strategies).•Each of the three layers must be represented at least ONCE across the five rows.

About Gypsies Step 1 Step 2 Step 3 Step 4

Metacog-nitive 

Thinking Session

List strategies you have used since the last 

interruption

Select ONE strategy you listed in Step 1 that was especially helpful in understanding this portion of the text.  What layer of reading is it?

Describe how you used this strategy WITH THIS TEXT.

Give the reason(s) why you chose this strategy for processing THIS PART OF THE TEXT. 

5 min to read and process 10 min to complete the four steps - >

1 

T.H.I.E.V.V.E.S. with Snatches

Superficial Reading &Telegram 

T.H.I.E.V.V.E.S. with Snatches

 BEFORE

I saw that the article had no title so I started to read the first sentence of each paragraph, but then cut this to reading only the first couple of lines.  As soon as I had a sense of the gist, I moved on.  I also read the last few lines of the article and was surprised that I could not find a conclusion.  I did two “snatches” along the way.I then decided to supplement the THIEVVES by doing a superficial read and pause at the end of each paragraph to do a Telegram.

Because there was no title, because text offered me nothing else besides prose text, THIEVVES was my best (only) option for previewing.  My purpose was to get the gist of the article—what it is about—so I didn’t need to read the entire first sentence.The two snatches definitely piqued my interestI read the last lines of the article but realized it was a huge sentence and was not a conclusion anyway.Doing all this helped to begin to get a sense of the text, --knew it was about the Romani and their language and migration patterns, and something about the connections between their language and the migrations.  I felt that THIEVVES had not provided quite enough information for me to write a good purpose and I also needed more information to try to figure out the main points and arguments.

5 min to read and process 10 min to complete the four steps - >

2

More Superficial ReadingMore TelegramLaunch Target, Track, & DefendKWLMy QuestionsAuthor on My Shoulder

TTD the Main Points DURING

 After I finished my superficial reading and making the telegrams on each sentence, I was able to formulate a purpose for reading this article.  But, I needed to full out the main arguments—thus TT&D.  I wrote my best guess and started to look for evidence for that main point about the one migration vs. several migrations.  I seemed to be doing just fine until I hit the last two paragraphs of the article which seemed to be pointing why the debate will continue; so, I revised my initial target.  Then I continued on to the end of article.  I feel I can defend the main points with the evidence I found. 

 Because I could tell there were several arguments in this debate about migrations, I decided to do Target, Track, and Defend the Main Points.  I wanted to clearly see the main points of each argument.  When I discovered a third argument, I was compelled to revise my target statement, which I did.  Those last two paragraphs had been problematic for me from the first THIEVVES.  What were they trying to say?  And because of TT&D I figured it ou.

Name______________________________________Date________ Section ____________Text__________________________

Performance Assessment of Metacognitive Thinking when Reading a Difficult Text

This draft of the rubric is still rough.  We are working to improve it.  Please do not use.

Self-Assess

Use the PAMA rubric to self-assess your metacognitive “performance” on the Melville paragraph.   NO ONE will see your results.

How did you feel doing this?   Why?

Student Reactions

They were asked an open-ended question:What did you think of the PAMA and how might it be improved?

A qualitative domain analysis was performed and a taxonomy created.

A Representative Sample of Student Reactions

The test showed me who I am as a reader:

I felt like this test was a confirmation to me that I really do know how

to apply reading strategies to a situation and use them effectively.

I found that I am able to read a passage and understand it no matter

how hard it is, or no matter the degree of my background knowledge.

This test helped me to realize that the strategies are now an integrated

part of my reading. No longer do I have look at the strategy to do it, I

can just do it. I can also integrate multiple strategies at once now.

The test elucidated the reading process for me:

• Primarily this test was revelatory for how much each of the reading strategies now influences my reading, and how related they are, how they influenced each other.

• I realized that some of the strategies I haven’t been using are more useful than I thought. The strategies that I do use are still really effective.

• I thought it helped me see where I can improve in reading—which is what an assessment should do!

• It revealed my level and capacity for critical thinking as I read.

Categories of Negative Student Reactions:• Number of Responses:  Have fewer/more response segments• Length of Time:  Shorten time for the test, lengthen it• Choice of article:  too easy, too hard, too short, not controversial

enough, didn’t like topic• Instructions:  not clear enough; wanted more class time to learn the

procedures and to know the rubric well beforehand.

Final Thoughts:• I think this is an excellent test because it requires us to use

knowledge and skills which we have been accumulating throughout the entire semester, as opposed to a typical final that requires the majority of the learning to take place a few minutes before the exam.

• If there’s any way to assess performance and understanding of these strategies, I believe this assessment is it. Very comprehensive as any test I’ve seen in a course. I highly recommend it!

Typical Professors’ Reactions

The PAMA seemed to be as authentic as possible in assessing the prime outcome we desired for our course.I saw totally engaged students.  They did not come up for air.  Several stopped by afterward to say this was intellectually stimulating, very enjoyable to do, and one of the best finals ever.I loved the activity, but scoring it was a chore—too much work for a final they won’t see.

PsychometricsValidity:

Ecological-related Evidence

Metacognitive Awareness is what we teach. From Day 1 

every preparation for class, every lesson, and 

everything students do as homework is a dress 

rehearsal for the PAMA.

Validity:Content-related Evidence

The PAMA taps into the purposes and assesses the desired outcomes of the course—To have developed metacognitive awareness and effective control of their reading processes when reading difficult academic texts is what the course is all about.   

Do students do the following tasks vital for processing these hard texts?• Do they recognize when comprehension breaks down?• Do they know what to do about it?• Do they know what, how, and why they are doing what they are 

doing as they read?

Reliability in Scoring:

Procedures  Trained raters  (TAs who had taken the course.) Raters scored a randomly selected set of protocols They re-scored the same set two weeks later

Results  of tests for intra and inter-rater reliability Pilot #2 – low reliability - too complex for raters  Pilot #3 – low reliability - still too complex for raters

Where from here?

Comments

Questions 

Suggestions

Experiences