TEMPLATE DESIGN © 2007
www.PosterPresentations.com
Aleahmad, T,. Aleven, V,. and Kraut, R. (in press) Open community authoring of worked example problems. In Proceedings of the International Conference of the Learning Sciences.
Aleahmad, T,. Aleven, V,. and Kraut, R. (under review) Open community authoring of targeted worked example problems . Proceedings of Intelligent Tutoring Systems 2008.
Anand, P. G., & Ross, S. M. (1987). Using Computer-Assisted Instruction to Personalize Arithmetic Materials for Elementary School Children. Journal of Educational Psychology, v79 n1 p72-78 Mar 1987.
Giles, J. (2005). Internet encyclopaedias go head to head, Nature, 438(7070), 900-901. doi: 10.1038/438900a.
Ku, H., & Sullivan, H. (2002). Student performance and attitudes using personalized mathematics instruction. Educational Technology Research and Development, 50(1), 21-34.
Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, pp. 98-129.
Schwonke, R., Wittwer, J., Aleven, V., Salden, R., Krieg, C., & Renkl, A. (2007). Can tutored problem solving benefit from faded worked-out examples. European Cognitive Science Conference, 23–27.
Important precondition for community-based authoring seems to be met: volunteers contribute high-quality math examples
Separating the wheat from the chaff required little effort
Both professional educators and amateurs contributed a large portion of useful materials.
Contributions from math teachers were not superior to those from others
Math teachers did write the best problem statements but amateurs wrote the best solutions
The student profile feature of the interface successfully drew out personalized resources
On every attribute the profile increased the likelihood of targeting it
Open community authoring of worked example problemsTuradg Aleahmad, Vincent Aleven, Robert Kraut
Human Computer Interaction Institute, Carnegie Mellon [email protected]
Abstract
Materials and Methods
Authoring Interface
Sample Submission
Quality of Open Access to Authoring
Literature Cited
Conclusions and Future Work
AcknowledgementsThanks to the ASSISTments project team.Thanks for Flickr user jenrock for photos used in personas.The research reported here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B040063 to Carnegie Mellon University.The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through "Effective Mathematics Education Research" program grant #R305K03140 to Carnegie Mellon University. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.
Open collaborative authoring systems such as Wikipedia are growing in use and impact. How well does this model work for the development of educational resources? In particular, will sufficient numbers of volunteers participate and contribute materials of high quality? Further, will they create resources that are targeted to students’ specific learning needs and, in order to engage students more, their personal characteristics? We conducted an experiment to explore these questions through the use of a novel prototype tool for community authoring of worked examples for a specific math skill. Participants were professional teachers (math and non- math) and amateurs. Participants were randomly assigned to the standard tool, or to an enhanced version that prompts authors to create materials for a specific (fictitious) student. We find that while there are differences by teaching status (math teacher, other teacher or amateur), all three groups made contributions of worth (as judged by two independent teacher raters) and that targeting a specific student leads contributors to author materials with greater potential to engage students. The experiment suggests that community authoring of educational resources is a feasible model of development and can enable new levels of personalization.
Effect of Student Profiles on Authoring
Mixed methodology experiment drew upon volunteers over the web to create worked example problems
Contributions were rated by machine and by two experts Three components of worked example evaluated by the
criteria below: Statement, Work and Explanation (alpha=0.61 for statement and 0.78 for all)
Numerical value
Rating category
Definition
0 Useless No use in teaching and it would be easier to write a new one than improve this one.
1 Easy fix Has some faults, but they are obvious and can be fixed easily, in under 5 minutes.
2 Worthy Worthy of being given to a student who matches on the difficulty and subject matter. Assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.
3 Excellent Excellent example to provide to some student. Again, assume that the system knows what's in the problem and what is appropriate for each student, based on their skills and interests.
Attribute With generic
(G)
With profiles not mentioning
attribute (N)
With profiles mentioning
attribute (M)
F-test(G-M)
F-test(N-M)
Female pronoun 5% 4% 16% 9.68* 12.82**Male pronoun 19% 14% 19% 0.004 1.19Sports word 9% 9% 24% 18.01** 11.89**TV word 4% 4% 10% 8.36* 2.63†Music word 2% 2% 9% 6.92* 8.93**Home word 14% n/a 20% 3.60* n/a
Count of submissions in each quality classification
Verbal skill shown
Sign. diffs
Mean reading level of
contribution
Std Err General math skill
shown
Sign.diffs
Probability of using 3-4-5
triangle
Std Err
High A 3.78 0.24 High A 16% 0.05Medium A B 3.56 0.32 Medium A B 26% 0.05Low B 2.93 0.33 Low B 27% 0.04GENERIC B 3.20 0.16 GENERIC A B 21% 0.03
Correspondence of verbal and math skill levels with the authoring interface
Example profile
Explanation of work
Illustrating solution steps
Illustrating solution
Problem statement
Submissions analyzed across teacher status of participant: math teacher, other teacher, or amateur
Motivation Work to solution Open collaborative development can work as well as
centralized development by experts (Gilles 2005) Traditional model-based tutors cost much time from skilled
experts (Murray 1999) Worked examples complement tutoring systems (Schwonke
et al. 2007). Personalization improves student engagement and test scores
(Ku and Sullivan 2002 , Anand and Ross 1987)
Of the raw submissions made, half were incompliant but also trivial to filter by simple automated methods
Of the remaining, a novice and a veteran teacher were able to rate each of them on three attributes in less than a minute each
About 1/10th were ready to help students learn without needing any modification
Statements were the highest quality components and solutions were the most difficult parts to author well
Math teachers were best at authoring problem statements
Amateurs authored the best worked solutions
Probabilities of authoring matching an attribute(†p<.10 *p<.05 **p<.001 )
All features of the profile display were accounted for in the problems submitted.
Discussing a male very much more likely than a female
Female profile brings likelihood to on par with males
High and low reading levels differed by almost a grade level
Submissions for profiles with high general math skill level were one third less likely to make use of simple 3-4-5 triangle problems
Experimental manipulation in which the subjects were presented with a student profile and told to provide instruction for that student
Math teachers Other teachers AmateursRegistered 131 170 1126Contributed also 70 72 428Passed vetting also 26 35 220