Page 1 of 1 NOTICE OF MEETING There will be a meeting of the ACADEMIC POLICY COMMITTEE Wednesday March 18, 2020 at 9:00am-11:00am In Room 209 Assumption Hall AGENDA Formal Business 1 Approval of Agenda 2 Minutes of meeting of February 12, 2020 3 Business arising from the minutes 4 Outstanding business Items for Approval 5 Reports/New Business 5.1 Timing Guidelines for Online Evaluations Antonio Rossini APC200318-5.1 5.2 Liberal Arts and Professional Studies (Aeronautics Leadership) Tamsin Bacon Admission Requirements – Revision APC200318-5.2 5.3 Policy on Undergraduate English Language Proficiency Requirement – Revision Beth Natale APC200318-5.3 APC200318-5.3a Item for Information 5.4 Office of Open Learning Annual Report (2018-2019) Nick Baker Lead Reader: Scott Martyn APC200318-5.4 Additional Business 6 Question period/Other business/Open Discussion 7 Adjournment Please carefully review the ‘starred’ (*) agenda items. As per the June 3, 2004 Senate meeting, ‘starred’ items will not be discussed during a scheduled meeting unless a member specifically requests that a ‘starred’ agenda item be ‘unstarred’, and therefore open for discussion/debate. This can be done any time before (by forwarding the request to the secretary) or during the meeting. By the end of the meeting, agenda items which remain ‘starred’ (*) will be deemed approved or received. APC200318A
Transcript
Page 1 of 1
NOTICE OF MEETING
There will be a meeting of the ACADEMIC POLICY COMMITTEE
Wednesday March 18, 2020 at 9:00am-11:00am In Room 209 Assumption Hall
AGENDA Formal Business 1 Approval of Agenda 2 Minutes of meeting of February 12, 2020 3 Business arising from the minutes 4 Outstanding business Items for Approval 5 Reports/New Business
5.1 Timing Guidelines for Online Evaluations Antonio Rossini APC200318-5.1
5.2 Liberal Arts and Professional Studies (Aeronautics Leadership) Tamsin Bacon Admission Requirements – Revision APC200318-5.2 5.3 Policy on Undergraduate English Language Proficiency Requirement – Revision Beth Natale APC200318-5.3
APC200318-5.3a
Item for Information 5.4 Office of Open Learning Annual Report (2018-2019) Nick Baker
Lead Reader: Scott Martyn APC200318-5.4
Additional Business 6 Question period/Other business/Open Discussion 7 Adjournment Please carefully review the ‘starred’ (*) agenda items. As per the June 3, 2004 Senate meeting, ‘starred’ items will not be discussed during a scheduled meeting unless a member specifically requests that a ‘starred’ agenda item be ‘unstarred’, and therefore open for discussion/debate. This can be done any time before (by forwarding the request to the secretary) or during the meeting. By the end of the meeting, agenda items which remain ‘starred’ (*) will be deemed approved or received.
APC200318A
Page 1 of 1
APC200318-5.1 University of Windsor
Academic Policy Committee 5.1: Timing Guidelines for Online Exams Item for: Approval Forwarded by: Senate MOTION: That APC establish a subcommittee, with the following membership, to consider timing guidelines for
online exams: Antonio Rossini Nick Baker 1-2 faculty members 1 student Rationale: At the February Senate meeting, concern was raised regarding the issue that students in online courses have a set time to write their exams which creates a barrier for students who are working. In light of this, students are requesting a broader window of time for writing exams for online courses, such as within a 24-hour time slot. In response, it was noted that some factors to consider include Senate bylaw provisions and academic integrity. Senate agreed that this item be forwarded to Academic Policy Committee for further review.
Page 1 of 2
APC200218-5.2 University of Windsor
Academic Policy Committee 5.2: Liberal Arts and Professional Studies (Aeronautics Leadership) Admission Requirements – Revision Item for: Approval Forwarded by: FAHSS MOTION: That the proposed revisions to the admission requirements for the Bachelor of Arts (Honours) in Liberal
Arts and Professional Studies (LAPS) – Aeronautics Leadership be approved. Proposed Revisions: [revisions are in bold] Minimum Admissions Average: 70% ENG4U and MHF4U, MCV4U, MDM4U or SPH4U Transport Canada Category 1 Medical Certificate Transport Canada Formal Aviation Language Proficiency Demonstration (Canadian citizens who have graduated from a Canadian English or French speaking high school are exempt) Supplementary Application Form: short essay/questions Rationale: • Both the Category 1 Medical Certificate and the Formal Aviation Language Proficiency Demonstration (ALPD) are
required by Transport Canada in order to work towards being a licenced commercial pilot in Canada. Canadian citizens who have graduated from a Canadian English or French speaking high school are exempt from the Formal ALPD. Currently, students complete these Transport Canada requirements just before or during the 1st year of the program. This has caused several problems: 1. The Medical Certificate: some students have not completed their medical certificate by the time they arrive on
campus to start the program. This delays their flight training, sometimes by months, if Transport Canada has a backlog of medicals to issue.
2. Language Proficiency: We have had students admitted to the program who really struggle with the English language. Even if they have passed University-required language testing, their understanding of English is sometimes still not strong enough to operate in the aviation world. Communication with flight instructors and air traffic control are absolute necessities in flight training and without that ability to communicate, it becomes a safety issue. When students are admitted to the program but cannot communicate effectively in English, they are not able to continue in the program.
• By requiring that the Category 1 Medical Certificate be completed before students have a confirmed offer, we will be able to identify students ahead of time who might have a medical issue that will keep them from flying. It will also minimize delays in flight training that occur when students have not completed the medical prior to starting the program.
• By requiring that the ALPD be completed before students have a confirmed offer, we will be able to identify students, prior to them beginning the program and spending thousands of dollars, who will not be able to start their flight training due to language issues.
• The addition of a supplementary application form will help in a couple of ways. First, there are only forty spots
Page 2 of 2
available in the first-year cohort. Up until now, these spots have been filled based on the earliest applications meeting the current admission requirements. In many ways this means the earlier a student applies, the better the chance they are admitted. For a program of this nature, that does not seem like the best way to assess suitability. In fairness, we have yet to fill all forty spots each year but as we get closer, we need a mechanism to assess suitability beyond how early a student applied. Secondly, we have received feedback from prospective students that they would like another opportunity to showcase what they can bring to the program, beyond their high school grades. The program is promoted as a leadership program and a supplementary essay/questionnaire can be used to ask students to discuss their past leadership experience or reasons they are interested in pursuing aviation/leadership education. This type of supplementary application is also used by other university aviation programs in Ontario.
Page 1 of 7
APC200318-5.3 University of Windsor
Academic Policy Committee 5.3: Policy on Undergraduate English Language Proficiency Requirement - Revision Item for: Approval Forwarded by: AVP, Enrolment Management MOTION: That the proposed revision to the Policy on Undergraduate English Language Proficiency
Requirements be approved. Proposed Revision [revisions are in bold] B. Undergraduate English Language Proficiency Requirement
All applicants, not covered by section A, must demonstrate proficiency in English to be considered for admission. This can be demonstrated in the following ways: […]
ADD: 8) A minimum score of 120 on the current Duolingo English Test (DET); or … […]
Background/Rationale: • Due to the Level 1 Public Health Emergency Response announced by the Provincial Governments in China related
to the coronavirus (COVID-19), TOEFL® and IELTS® testing in Mainland China has been suspended through the end of March
• In light of recent temporary closures of language proficiency test centres in China we would like to provide an online method for students to demonstrate English competency
• The Duolingo English Test (DET) is an English language proficiency test conducted online. The adaptive test evaluates reading, writing, speaking and listening skills and a video interview assesses student’s responses to open-ended question. Each test taker's proficiency is reported as a holistic score on a scale from 10 – 160 (see Appendix I)
• DET is a computer adaptive test with results highly correlated to other major assessments, including TOEFL® and IELTS®
• The proposed minimum score of 120 on the DET has a tested concordance to other standardized tests including TOEFL® of 97 – 102, IELTS® 7.0, and Common European Framework of Reference for Languages (CEFR) C1 (see: https://englishtest.duolingo.com/research)(see Appendix II)
• DET is accepted by over 500 institutions, including Bishop’s, Concordia, McGill, Mount Allison, and St. Clair, plus Arizona State, UCLA, University of Colorado Boulder, University of Illinois, NYU, and Washington U (see Appendix III)
• A quick video about how a student uploads their ID for validation as well as other resources about the test can be found at: https://englishtest.duolingo.com/resources
• The proposal has been reviewed and supported by the Office of the Registrar, Student Recruitment, and the Centre for English Language Development
Page 2 of 7
Appendix 1. The Duolingo English Test is designed to assess real-world language ability, by measuring reading, writing, speaking, and listening skills. Each test taker's proficiency is reported as a holistic score on a scale from 10 – 160.
Page 3 of 7
Appendix II. Based on the research that links Duolingo English Test scores to TOEFL iBT total scores and IELTS overall band scores, Duolingo has developed the following comparison table to help you make well-informed admissions decisions. For more information see: https://englishtest.duolingo.com/scores
Data is based on 2,319 individual test takers who took the Duolingo English Test and TOEFL iBT (correlation .77) and 991 individual test takers who took both the Duolingo English Test and IELTS Academic (correlation .78). These correlation coefficients are considered a strongly positive statistical relationship for comparing scores. Learn more about score reliability on our research page.
Page 4 of 7
Appendix III. Select examples of Canadian Institutions that accept the Duolingo English Language Test for Undergraduate Admissions purposes. For a complete list see: https://englishtest.duolingo.com/institutions Universities
Institution Min. Score (UGrad)
Link
Bishop’s University 120 Concordia University 120 Click here McGill University 115 Click here Quest University 110 Click here St. Thomas University 105 Click here
Colleges
Institution Min. Score (UGrad)
Link
New Brunswick Community College 100 Click here Vancouver Film School 105 Click here
Notable US Universities, include:
• Arizona State University • Baylor University • Columbia University • Dartmouth College • Duke University • Iowa State University • Michigan State University • New York University • Stanford University • University of Chicago • University of San Francisco • Vanderbilt University • Yale University
Page 5 of 7
Appendix IV. Proposed modifications to the “Policy on Undergraduate English Language Proficiency Requirement” … B. Undergraduate English Language Proficiency Requirement All applicants, not covered by section A, must demonstrate proficiency in English to be considered for admission. This can be demonstrated in the following ways:
1) A minimum score of 6.5 on the International English Language Testing System (IELTS); or 2) A minimum score of 60 on the Carleton University’s Canadian Academic English Language Assessment
(CAEL); or 3) A minimum score of 60 on the Pearson Test of English Academic (PTE); or 4) A minimum overall score of 176 on the Cambridge English: Advanced (CAE) or the Cambridge
English:Proficiency (CPE); or 5) Successful completion of the University of Windsor's English Language Improvement Program (ELIP)* (level
III) with a minimum final grade of 75%; or 6) A minimum score of 85 on the English Proficiency Test administered by the English Language Institute of
the University of Michigan (MELAB); or 7) A minimum score of 83 with an writing score of 20 on the Internet-based Test (IBT) of English as a Foreign
Language (TOEFL). Prospective students completing the Paper-Delivered Test (PDT) of English as a Foreign Language (TOEFL) (as offered from October 2017) will have their results assessed on an individual basis; or
8) A minimum score of 120 on the current Duolingo English Test (DET); or 9) Receive a minimum of five semester transfer credits based on work completed at a recognized English-
speaking University. (Normally, students must have at least one term of full-time study.); or 10) Receive a minimum of one year of transfer credit based on work completed at a non-university post-
secondary institution with an overall transfer average of B; or 11) Have completed all secondary or post-secondary education in English speaking countries such as the
United Kingdom, West Indies and other countries as specified by the Office of the Registrar; or 12) Can demonstrate that they have completed three years of full-time secondary or post-secondary study in a
school where the language of instruction is English (e.g., in a British, Canadian, American or Australian Curriculum or an international diploma like the International Baccalaureate) and can demonstrate English proficiency. In some cases an English proficiency test may still be required; or
13) Applicants who do not satisfy the above requirements and wish exemption must provide the Office of the Registrar with a letter detailing the reasons for their appeal and include a letter of recommendation from their Secondary School principal or designate. The Registrar (or designate) will assess the request for exemption of this requirement.
Page 6 of 7
Appendix V. Duolingo Applicant Flyer
Page 7 of 7
Appendix VI. Duolingo Institutional Proposal See APC200319-5.3a for Duolingo English Test: Technical Manual
Duolingo English Test:Technical Manual
Duolingo Research ReportJuly 15, 2019 (25 pages)https://englishtest.duolingo.com/research
Geoffrey T. LaFlair∗ and Burr Settles∗
Abstract
The Duolingo English Test Technical Manual provides an overview of the design, development,administration, and scoring of the Duolingo English Test. Furthermore, it reports on test takerdemographics and the statistical characteristics of the test. This is a living document and will beupdated regularly (last update: July 15, 2019).
The Duolingo English Test is a measure of English language proficiency. The test has beendesigned for maximum accessibility; it is delivered via the internet, without a testing center,and is available 24 hours a day, 365 days a year. It has been designed to be efficient. Ittakes less than one hour to complete the entire process of taking the test (i.e., onboarding, testadministration, uploading). It is a computer-adaptive test (CAT), and it uses item types thatprovide maximal information about English language proficiency. It is designed to be user-friendly; the onboarding, user interface, and item formats are easy to interact with.This document provides an overview of the design the Duolingo English Test. It contains a
discussion of:
• the test’s accessibility, delivery, proctoring and security processes;• the demographic information of the test taking population;• the test’s items, how they were created, and how they are are delivered and scored;• and the statistical characteristics of the Duolingo English Test.
The test scores are intended to be interpreted as reflecting test takers’ English language abilityand used in a variety of settings, including for university admissions decisions.
2 Accessibility
Broad accessibility is one of the central motivations for the Duolingo English Test. Testsadministered at test centers consume resources which limit accessibility: they requireappointments at a physical testing center within certain hours on specific dates (and travelto the test center), and carry considerable registration fees. The maps in Figure 1 show theconcentration of test centers in the world (left panel) compared to internet penetration in theworld (right panel). The figure shows a stark difference in how much more easily an internet-based test can be accessed than a test center*. While the ratio of internet access to test centeraccess is a somewhat limited metric — not every internet user has access to a device that canrun the Duolingo English Test, and physical test centers can usually handle dozens of test-takersat once — it is still clear that the potential audience for the Duolingo English Test is orders ofmagnitude larger than those who could be served currently by more traditional test centers. Bydelivering assessments on-demand, 24 hours a day, to an estimated 2 billion internet-connecteddevices anywhere in the world for US$49, we argue that the Duolingo English Test holds thepotential to be the most accessible valid and secure language assessment platform in the world.
Figure 1. Global heatmaps comparing the number of physical test centers (left) per 1 million inhabitants(TOEFL®, IELTS®, and PTE® combined) vs. the percentage of internet users (right).
3 Test Administration and Security
The Duolingo English Test is administered online, via the internet to test takers. The security ofDuolingo English Test scores is ensured through a robust and secure onboarding process, rulesthat test takers must adhere to during the test administration, and a strict proctoring process.All of these procedures are evaluated after the test has been administered and prior to scorereporting.
3.1 Test Administration
Test takers are required to take the test alone in a quiet environment. The Duolingo English Testcan be taken in the Chrome and Opera browsers worldwide. In China, the test can be taken onthe the 360 and QQ browsers as well. An internet connection with at least 2 Mbps downloadspeed and 1 Mbps upload speed is recommended for test sessions.
3.2 Onboarding
Before the test is administered, test takers complete an onboarding process. This processchecks that the computer’s microphone and speaker work. It is also at this time that the testtakers’ identification information is collected and that test takers are informed of the test’sadministration rules.
3.3 Administration Rules
The list behaviors that are prohibited during an administration of the Duolingo English Test arelisted below. In addition to these behavioral rules, there are rules for the test takers’ internetbrowsers. The browsers are locked down after onboarding, which means that any navigationaway from the browser invalidates the test session. Additionally, all browser plugins must bedisabled.
• Interacting with anyone• Allowing other people in the room• Using headphones or earbuds• Disabling the microphone or camera
• Looking off screen• Moving out of frame of the camera• Accessing any outside material or devices• Leaving the web browser
3.4 Proctoring & Reporting
After the test has been completed and uploaded, it undergoes a thorough proctoring review usinghuman proctors with TESOL/applied linguistics expertise, which is supplemented by artificialintelligence to call proctors’ attention to suspicious behavior. This process take no more than48 hours after the test has been uploaded. After the process has been completed, score reportsare sent electronically to the test taker and any institutions they elect to share their scores with.Test takers can share their scores with an unlimited number of institutions.
4 Test Taker Demographics
In this section, test taker demographics are reported. During the onboarding process of each testadministration, test takers are asked to report their first language (L1), date of birth, and theirgender identity. Their country of residence is loggedwhen they show their proof of identificationduring the onboarding process. There were 23,460 people who took certified Duolingo EnglishTests between August 1, 2017 and June 30, 2019.The most frequent L1s of Duolingo English Test test takers include Mandarin, Spanish,
Arabic, and Portuguese (see Table 1). Duolingo English Test test takers represent 115 uniqueL1s and 168 unique countries. The full tables of all test taker L1s and countries of origin can befound in the Appendix.
Table 1. Most Frequent Test Taker L1s
First Language
Chinese - MandarinSpanishArabicPortugueseEnglishKoreanJapaneseFrenchRussianKurdish
Reporting gender identity during the onboarding process is optional, but reporting birth dateis required. Table 2 shows that 34.60% of Duolingo English Test test takers identified as female,41.29% of test takers identified as male, and 24.08% chose not to report. Table 3 shows that78% of Duolingo English Test test takers are between 16 and 30 years of age.
The test has seven different item types, which collectively tap into test takers’ reading, writing,listening, and speaking abilities. Because the Duolingo English Test is a CAT, it will adjust indifficulty as the computer updates its real-time estimate of test takers’ language proficiency asthey progress through the test. There are five item types in the computer-adaptive portion ofthe test. The CAT item types include c-test, audio yes/no vocabulary, visual yes/no vocabulary,dictation, and elicited imitation. During each administration of the Duolingo English Test, atest taker will see at minimum three of each CAT item type and at maximum of seven of eachCAT item type. The median rate of occurrence of the CAT item types across all administrationsis six times per test administration. Additionally, test takers respond to four writing promptsand three speaking prompts. They are not a part of the computer-adaptive portion of the test.However, the writing and speaking prompts also vary in difficulty, and their selection is basedon the CAT’s estimate of test taker ability. These items work together to measure test takers’English language proficiency in reading, writing, listening, and speaking.
5.1 C-test
The c-tests provide a measure of the test takers’ reading ability (Klein-Braley 1997; Khodadady2014). In this task, the first and last sentences are fully intact, while words in the interveningsentences are “damaged” by deleting the second half of the word. Test takers respond to thec-test items by completing the damaged words in the paragraph (see Figure 2). The test takerneeds to rely on context and discourse information to reconstruct the damaged words (whichspan multiple vocabulary and morpho-syntactic categories). It has been shown that c-tests are
significantly correlated with many other major language proficiency tests, and additionally arerelated to spelling skills (Khodadady 2014).
5.2 Yes/No Vocabulary
This is a variant of the “yes/no” vocabulary test (Beeckmans et al. 2001). The test taker ispresented with a set of English words mixed with pseudo-words that are designed to appearEnglish-like, and must discriminate between them.† Such tests have been used to assessvocabulary knowledge at various CEFR levels (Milton 2010), and have been shown to predictlanguage proficiency skills—the text version (see top panel in Figure 3) predicts listening,reading, and writing abilities; while the audio version (see bottom panel in Figure 3) predictslistening and speaking abilities in particular (Milton, Wade, and Hopkins 2010; Staehr 2008).These tests typically show a large set of stimuli (e.g., 60 words and 40 pseudo-words) of mixeddifficulty at once. The format is made computer-adaptive by successively presenting multiplesets (items/testlets), each containing a few stimuli of the same difficulty (e.g., B1-level wordswith pseudo-words that should be B1-level if they existed; more on how this is done in Section6.1).
5.3 Dictation
In this exercise, the test taker listens to a spoken sentence or short passage and then transcribesit using the computer keyboard‡ (see Figure 4). The test takers have one minute in total to listento and transcribe what they heard. They can play the passage up to three times. This assesses
†We use an LSTM recurrent neural network trained on the English dictionary to create realistic pseudo-words, filteringout any real words, acceptable regional spellings, and pseudo-words that orthographically or phonetically resemble realEnglish words too closely.‡Autocomplete, spell-checking, and other assistive device features or plugins are detected and disabled.
test taker ability to recognize individual words and to hold them in memory long enough toaccurately reproduce them; both are critical for spoken language understanding (Bradlow andBent 2002; Buck 2001; Smith and Kosslyn 2007). Dictation tasks have also been found to beassociated with language learner intelligibility in speech production (Bradlow and Bent 2008).
5.4 Elicited Imitation (Read-aloud)
The read-aloud variation of the elicited imitation task—example in Figure 5—is a measureof test taker reading and speaking abilities (Vinther 2002; Jessop, Suzuki, and Tomita 2007;Litman, Strik, and Lim 2018). It requires the test takers to read, understand, and speak asentence. Test takers respond to this task by using the computer’s microphone to recordthemselves speaking a written sentence. The goal of this task is to evaluate intelligiblespeech production, which is affected by segmental/phonemic and suprasegmental properties
like intonation, rhythm, and stress (Anderson-Hsieh, Johnson, and Koehler 1992; Derwing,Munro, and Wiebe 1998; Field 2005; Hahn 2004). Furthermore, intelligibility is correlatedwith overall spoken comprehensibility (Munro and Derwing 1995; Derwing and Munro 1997;Derwing, Munro, and Wiebe 1998), meaning that this item format can capture aspects ofspeaking proficiency. We use state-of-the-art speech technologies to extract features of spokenlanguage, such as acoustic and fluency features that predict these properties (in addition to basicautomatic speech recognition), thus evaluating the general clarity of speech.
The extended speaking tasks are measures of test taker English speaking abilities. At the endof the CAT portion of the test, the test takers respond to four speaking prompts: one picturedescription task and three independent speaking tasks, two with a written prompt and one withan aural prompt (see Figure 6). Each of the task types have items that are calibrated for high,intermediate, and low proficiency levels. The difficulty level of the tasks that test takers receiveis conditional on their estimated ability in the CAT portion of the test. All of these task typesrequire the test taker to speak for an extended time period and to leverage different aspectsof their organizational knowledge (e.g., grammar, vocabulary, text structure) and functionalelements of their pragmatic language knowledge (e.g., ideational knowledge) (Bachman andPalmer 1996).
5.6 Extended Writing
The extended writing tasks are measures of the test takers writing English abilities. Test takersrespond to four writing prompts that require extended responses: three picture description tasksand one independent task with a written prompt (see Figure 7). Similar to the speaking tasks,these are drawn from different levels of difficulty conditional on the estimated ability level of thetest taker. The stimuli in the picture description tasks were selected by people with graduate-level degrees in applied linguistics. They are designed to give test takers the opportunity todisplay their full range of written language abilities (Cushing-Weigle 2002). The independenttasks require test takers to describe, recount, or make an argument; these require the test takers todemonstrate more discursive knowledge of writing in addition to language knowledge (Cushing-Weigle 2002).
6 Development, Delivery, & Scoring
This section explains how the computer-adaptive items in the test were developed, howthe computer-adaptive test works, and how the items are scored. Additionally, it providesinformation about the automated scoring systems for the speaking and writing tasks and howthey were evaluated.
6.1 Item Development
In order to create enough items of each type at varying levels of difficulty, the Duolingo EnglishTest item pool is automatically generated (and very large). As a result, it is not feasible toestimate bi (item difficulty) statistically from actual administrations due to data sparsity, and it isnot scalable to have each itemmanually reviewed by CEFR-trained experts. Instead, we employstatistical machine learning (ML) and natural language processing (NLP) to automaticallyproject items onto the Duolingo English Test scale. Each of the items has an estimated levelof difficulty on a continuous scale between zero and ten. These levels were assigned to theitems based on one of two ML/NLP models—a vocabulary model and a passage model—thatwere trained as part of the test development process. The vocabulary model was used to estimatethe item difficulty of the yes/no vocabulary tasks. The passage model was used to estimate thedifficulty of the other item types. The two models are used to predict b̂i values for the different
Once items are generated, calibrated (b̂i estimates are made), and placed in the item pool, theDuolingo English Test uses CAT approaches to administer and score tests (Wainer 2000; Segall2005). Because computer-adaptive administration gives items to test takers conditional on theirestimated ability, CATs have been shown to be shorter (Thissen and Mislevy 2000) and provideuniformly precise scores for most test takers when compared to fixed-form tests (Weiss andKingsbury 1984).To do this, we employ a generalization of item response theory (IRT). The conditional
probability of an observed item score sequence g = ⟨g1, g2, . . . , gt⟩ given θ is the product of allthe item-specific item response function (IRF) probabilities (assuming local item independence):
p(g|θ) =t∏
i=1
pi(θ)gi(1− pi(θ))
1−gi , (1)
where gi denotes the scored response to item i (typically gi = 1 if correct, gi = 0 ifincorrect), and 1− pi(θ) is the probability of an incorrect response under the IRF model. Animplication of local independence is that the probability of responses for two separate test itemsi and j are independent of each other, controlling for the effect of θ.The purpose of a CAT is to estimate the ability (θ) of test takers as precisely as possible with
as few test items as possible. The precision of our θ estimate depends on the item sequence g:test takers of higher ability θ are best assessed by items with higher difficulty bi (and likewisefor lower values of θ and bi). The true value of a test taker’s ability (θ) is unknown beforetest administration. As a result, an iterative adaptive algorithm is required. First, the algorithmmakes a provisional estimate of θ̂t based on responses to a set of items at the beginning of thetest — increasing in difficulty — to time point t. Then the difficulty of the next item is selectedas a function of the current estimate: bt+1 = f(θ̂t). Once that item is scored and added to g,the process repeats until a stopping criterion is satisfied.The maximum-likelihood estimation (MLE) approach to finding θ̂t and selecting the next
item is based on the log-likelihood function:
LL(θ̂t) = logt∏
i=1
pi(θ̂t)gi(1− pi(θ̂t))
1−gi
=
t∑i=1
gi log pi(θ̂t) + (1− gi) log(1− pi(θ̂t)). (2)
The first line directly follows from Equation (1), and is a typical formulation in the IRTliterature. The rearrangement on the second line more explicitly relates the objective tominimizing cross-entropy (de Boer et al. 2005), a measure of disagreement between twoprobability distributions. This is because our test items are scored probabilistically (see Section6.3. As a result, gi is a probabilistic response (0 ≤ gi ≤ 1) rather than a binary response(gi ∈ {0, 1}). The MLE optimization in Equation (2) seeks to find the θ̂t that yields an IRFprediction pi(θ̂t) that is most similar to each scored response gi ∈ g. This generalization,
combined with concise and predictive item formats, helps to minimize test administration timesignificantly.Duolingo English Tests are variable-length, meaning that exam time and number of items
can vary with each administration. The iterative adaptive procedure continues until either thevariance of the θ̂t estimate drops below a certain threshold, or the test exceeds amaximum lengthin terms of minutes or items. Most tests are less than 30-45 minutes long (including speakingand writing; excluding onboarding and uploading), and the median test consists of about 27computer-adaptive (and eight extended response items) items with over 200 measurements§
Once the algorithm converges, the final reported score is not the provisional MLE point-estimate given by Equation (2) used during CAT administration. Rather, p(θ|g) is computedfor the CAT items for each possible θ ∈ [0, 10] and normalized into a posterior distribution inorder to create a weighted average score for each item type. These weighted average scores ofeach CAT item type are then used to create a composite score with the scores of the speakingand writing tasks.
6.3 Item Scoring
All test items are scored automatically via statistical procedures developed specifically for eachformat. For example, the yes/no vocabulary (see Figure 3) format is traditionally scored usingthe sensitivity index d′: a measure of separation between signal (word) and noise (pseudo-word)distributions from signal detection theory (Beeckmans et al. 2001, Zimmerman et al. 1977).However, traditional yes/no tests assume that all stimuli are given at once, which is not the casein Duolingo English Test’s adaptive variant. This index, d′, is easily computed for fewer stimuli,and it has a probabilistic interpretation under receiver-operator characteristics (ROC) analysis(Fawcett 2006). That is, d′ is calculated for each test taker by item response and converted it to ascore gi, which can be interpreted as “the test taker can accurately discriminate between Englishwords and pseudo-words at this score/difficulty level with probability gi,” where gi ∈ [0, 1].Similarly, the responses to the dictation, elicited imitation, and c-test tasks are aligned against
an expected reference text, and similarities and differences in the alignment are evaluated.The output of the comparison is used in a (binary) logistic regression model¶ to provide itsprobabilistic score gi.
6.4 Extended Speaking and Writing Tasks
The writing and speaking tasks are scored by automated scoring algorithms developed by MLand NLP experts at Duolingo. There are two separate algorithms: one for the speaking tasksand one for the writing tasks. Currently, the scores for the tasks are estimated at the portfoliolevel—meaning that the speaking score that is included in the composite score represents the testtakers’ performance on the four speaking tasks and the writing score represents the test takers’performance on the four writing tasks.
§For example, each word (or pseudo-word) in the vocabulary format, and each damaged word in the c-test passageformat, is considered a separate “measurement” (or sub-item).¶The weights for this model were trained on aggregate human judgments of correctness and intelligibility on tens ofthousands of test items. The correlation between model predictions and human judgments is r =~0.75 (p < 0.001).
The writing scoring algorithm was trained on 3,626 writing performances, and the speakingscoring algorithmwas trained on 3,966 performances. Both sets of performances were scored byby human raters with TESOL/applied linguistics training. The algorithms were then evaluatedthrough a process known as cross-validation. In this process, they are trained on a portion of thedata (90%; the training set) and then evaluated on the remaining portion (10%; the test set). Thisdesign is called 10-fold cross-validation because the analysis is repeated 10 times on differentconfigurations of 90/10 training/test sets.This analysis used Cohen’s κ as the index of agreement (results in Table 4). It is a measure
of probability of agreement with chance agreement factored out. The first row shows the rateof human-human agreement. The last two rows show rates of human-machine agreement. Theκ index reflects agreement when the training set is used as the test set; this is expected to behigher than the cross-validated analysis. The κxv index shows the rates of agreement whenusing cross-validated analysis. All human-machine relationships show high rates of agreement(κ > 0.70) between the algorithms’ scores and human rater scores.
This section provides an overview of the statistical characteristics of the Duolingo English Test.It includes information about the total score distribution and three reliability measures: test-retest, internal consistency, and standard error of measure.
7.1 Score Distribution
The following reports on an analysis of the composite scores for test administered betweenAugust 1, 2017 to June 30, 2019 that have been rescored to reflect the current operational scale:10-160 in five point increments. This is the operational scale for all tests administered on orafter July 15, 2019.
Figure 8 shows the distribution of test scores on the 10-160 point scale (on the x-axis of eachplot). The top panel shows the empirical cumulative density function (ECDF) of the test scores.Where a test score meets the line in the ECDF, it shows the proportion of scores at or belowthat point. The middle panel shows the density function of the test scores, and the bottom panelshows a box plot of the total test scores.The plots in Figure 8 show some negative skew, which is reflected in the descriptive statistics
in Table 5. The mean and the median test score are 98.87 and 100 respectively, and theinterquartile range is 30. Table 9 in the Appendix shows the percentage and cumulativepercentage of test takers at each score point.
The reliability of the Duolingo English Test is evaluated by examining the relationship betweenscores from repeated test sessions (test-retest reliability), the relationship among the differenthalves of each test (split-half reliability; a measure of internal consistency), and the standarderror of measure (SEM). The SEM is the range of test scores in which the test takers true scoreexists. It is a function of the test’s reliability and standard deviation as shown in Equation (3).
SEM = SD ∗√1− reliability (3)
There have been 776 test takers who have taken the test twice within 30 days. The correlationbetween test scores at time one and test scores at time two is strong, positive, and significant (r =0.85, p < 0.001). The internal consistency of the Duolingo English Test is evaluated using split-half methods on the computer-adaptive portion only. A representative balance item types oneach half is controlled for. The split-half reliability of the Duolingo English Test is also strong,positive, and significant (n = 23,460, r = 0.91, p < 0.001). Using the test-retest reliabilitycoefficient results in a conservative estimate of the SEM (+/- 8.71 score points), which meansthat a test taker’s true score falls within a range of 8.71 score points above or below theirestimated ability.
7.3 Relationship with Other Tests
While the Duolingo English Test is being uploaded after the test administration, we ask testtakers to share their recent TOEFL iBT and IELTS test scores (data collection started in August2018). This data was used to evaluate the relationship between these two tests and the DuolingoEnglish Test and to create concordance tables for the total test scores.
Correlation Pearson’s correlations coefficients were estimated to evaluate the relationshipbetween the Duolingo English Test and the TOEFL iBT and IELTS. The correlation coefficientsfor both revealed strong, positive, and significant relationships between the Duolingo EnglishTest scores and the TOEFL iBT scores (n = 2,319, r = 0.77, p < 0.001) and IELTS scores (n =991, r = 0.78, p < 0.001). These relationships are visualized in Figure 9. The left panel showsthe relationship between the Duolingo English Test and TOEFL iBT, and the right panel showsthe relationship between the Duolingo English Test and IELTS.
Concordance The same data from the correlation study was used to create concordance tablesfor Duolingo English Test users. Two types of equating were compared: equipercentile (Kolenand Brennan 2014) and kernel equating (Davier, Holland, and Thayer 2003). Within eachequating type two methods were evaluated: 1) loglinear pre-smoothing that preserved the firstand second moments as well as the bivariate relationship between the test scores and 2) loglinearpre-smoothing that preserved the first, second, and third moments as well as the bivariaterelationship between the test scores. The equate (Albano 2016) and kequate (Andersson,Bränberg, and Wiberg 2013) packages in R (R Core Team 2018) were used to conduct theequating study.
The equating procedure that was selected to create the concordance tables was the one thatminimized the mean standard error of equating. Table 6 shows that the the kernel equating thatpreserved the first two moments and the bivariate score relationship introduces the least amountof error. Figure 10 shows that the conditional error across the Duolingo English Test score rangeis very small for kernel equating as well. As a result, we used the “KER 2” equating relationship
to create our concordance tables. The concordance tables can be found at the Duolingo EnglishTest scores page (https://englishtest.duolingo.com/scores).
8 Conclusion
The research reported here illustrates evidence for the validity of the interpretations and uses ofthe Duolingo English Test. Updated versions of this document will be released as we continueour research.
Albano, A. D. 2016. “equate: An R Package for Observed-Score Linking and Equating.”Journal of Statistical Software 74 (8): 1–36. https://doi.org/10.18637/jss.v074.i08.
Anderson-Hsieh, J., R. Johnson, and K. Koehler. 1992. “The Relationship Between NativeSpeaker Judgments of Nonnative Pronunciation and Deviance in Segmentals, Prosody, andSyllable Structure.” Language Learning 42: 529–55.
Andersson, Björn, Kenny Bränberg, and Marie Wiberg. 2013. “Performing the Kernel Methodof Test Equating with the Package kequate.” Journal of Statistical Software 55 (6): 1–25.http://www.jstatsoft.org/v55/i06/.
Bachman, L. F., and A. S. Palmer. 1996. Language Testing in Practice. Oxford: OxfordUniversity Press.
Beeckmans, R., J. Eyckmans, V. Janssens, M. Dufranne, and H. Van de Velde. 2001.“Examining the Yes/No Vocabulary Test: Some Methodological Issues in Theory andPractice.” Language Testing 18 (3): 235–74.
Bradlow, A. R., and T. Bent. 2002. “The Clear Speech Effect for Non-Native Listeners.”Journal of the Acoustical Society of America 112: 272–84.
———. 2008. “Perceptual Adaptation to Non-Native Speech.” Cognition 106: 707–29.Buck, G. 2001. Assessing Listening. Campbridge: Cambridge University Press.Cushing-Weigle, S. 2002. Assessing Writing. Cambridge: Cambridge University Press.Davier, Alina A von, Paul W Holland, and Dorothy T Thayer. 2003. The Kernel Method of
Test Equating. NY: Springer Science & Business Media.
de Boer, P. T., D. P. Kroese, S. Mannor, and R. Y. Rubinstien. 2005. “A Tutorial on the Cross-Entropy Method.” Annals of Operations Research 34: 19–67.
Derwing, T. M., and M. J. Munro. 1997. “Accent, Intelligibility, and Comprehensibility:Evidence from Four L1s.” Studies in Second Language Acquisition 19 (1): 1–16.
Derwing, T. M., M. J. Munro, and G. Wiebe. 1998. “Evidence in Favor of a Broad Frameworkfor Pronunciation Instruction.” Language Learning 48: 393–410.
Fawcett, T. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8):861–74.
Field, J. 2005. “Intelligibility and the Listener: The Role of Lexical Stress.” TESOL Quarterly39: 399–423.
Hahn, L. D. 2004. “Primary Stress and Intelligibility: Research to Motivate the Teaching ofSuprasegmentals.” TESOL Quarterly 38: 201–23.
Jessop, L., W. Suzuki, and Y. Tomita. 2007. “Elicited Imitation in Second Language AcquisitionResearch.” Canadian Modern Language Review 64 (1): 215–38.
Khodadady, E. 2014. “Construct Validity of C-tests: A Factorial Approach.” Journal ofLanguage Teaching and Research 5 (November).
Klein-Braley, C. 1997. “C-Tests in the Context of Reduced Redundancy Testing: AnAppraisal.”Language Testing 14 (1): 47–84.
Kolen, Michael J, and Robert L Brennan. 2014. Test Equating, Scaling, and Linking: Methodsand Practices. NY: Springer Science & Business Media.
Litman, D., H. Strik, and G. S. Lim. 2018. “Speech Technologies and the Assessment of SecondLanguage Speaking: Approaches, Challenges, and Opportunities.” Language AssessmentQuarterly, 1–16.
Milton, J. 2010. “The Development of Vocabulary Breadth Across the CEFR Levels.” InCommunicative Proficiency and Linguistic Development: Intersections Between SLAand Language Testing Research, edited by I. Bartning, M. Martin, and I. Vedder, 211–32.Eurosla.
Milton, J., J. Wade, and N. Hopkins. 2010. “Aural Word Recognition and Oral Competence inEnglish as a Foreign Language.” In Insights into Non-Native Vocabulary Teaching andLearning, edited by R. Chacón-Beltrán, C. Abello-Contesse, and M. Torreblanca-López,52:83–98. Bristol: Multilingual Matters.
Munro, M. J., and T.M. Derwing. 1995. “Foreign Accent, Comprehensibility, and Intelligibilityin the Speech of Second Language Learners.” Language Learning 45: 73–97.
R Core Team. 2018. R: A Language and Environment for Statistical Computing. Vienna,Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Segall, D. O. 2005. “Computerized Adaptive Testing.” In Encyclopedia of SocialMeasurement, edited by K. Kempf-Leonard. New York, NY: Elsevier.
Settles, B., M. Hagirawa, and G. T. LaFlair. under revision. “Machine Learning DrivenLanguage Assessment.” Transactions in Applied Computational Linguistics.
Smith, E. E., and S. M. Kosslyn. 2007. Cognitive Psychology: Mind and Brain. Upper SaddleRiver, NJ: Pearson/Prentice Hall.
Staehr, L. S. 2008. “Vocabulary Size and the Skills of Listening, Reading and Writing.”Language Learning Journal 36: 139–52.
Thissen, D., and R. J. Mislevy. 2000. “Testing Algorithms.” In Computerized AdaptiveTesting: A Primer, edited by H. Wainer. Routledge.
Vinther, T. 2002. “Elicited Imitation: A Brief Overview.” International Journal of AppliedLinguistics 12 (1): 54–73.
Wainer, H. 2000. Computerized Adaptive Testing: A Primer. Hillsdale, NJ: Routledge.Weiss, D. J., and G. G. Kingsbury. 1984. “Application of Computerized Adaptive Testing to
Educational Problems.” Journal of Educational Measurement 21: 361–75.
Zimmerman, J., P. K. Broder, J. J. Shaughnessy, and B. J. Underwood. 1977. “A RecognitionTest of Vocabulary Using Signal-Detection Measures, and Some Correlates of Word andNonword Recognition.” Intelligence 1 (1): 5–31.
Afrikaans Dutch Italian Mandingo SwahiliAkan Efik Japanese Marathi SwedishAlbanian English Kannada Mongolian TagalogAmharic Ewe Kashmiri Nepali TajikArabic Farsi Kazakh Northern Sotho Tamil
Armenian Finnish Khmer Norwegian TatarAzerbaijani French Kikuyu Oriya TeluguBambara Fulah Kinyarwanda Oromo ThaiBelarusian Ga Kirundi Polish TibetanBemba Galician Kongo Portuguese Tigrinya
Bengali Ganda Konkani Punjabi TswanaBikol Georgian Korean Pushto TurkishBosnian German Kurdish Romanian TurkmenBulgarian Greek Lao Russian TwiBurmese Guarani Latvian Serbian Uighur
Catalan Gujarati Lingala Sesotho UkrainianCebuano Hausa Lithuanian Shona UrduChichewa (Nyanja) Hebrew Luo Sinhalese UzbekChinese - Cantonese Hindi Luxembourgish Slovak VietnameseChinese - Mandarin Hungarian Macedonian Slovenian Wolof
Croatian Icelandic Malagasy Somali XhosaCzech Igbo Malay Spanish YorubaDanish Indonesian Malayalam Sundanese Zulu
Table 8. Test Taker Country Origins in Alphabetical Order
Afghanistan Cyprus Lebanon Russian FederationAlbania Czechia Lesotho RwandaAlgeria Denmark Liberia Saudi ArabiaAmerican Samoa Dominica Libya SenegalAngola Dominican Republic Lithuania Serbia
Antigua and Barbuda Ecuador Luxembourg Sierra LeoneArgentina Egypt Macao SingaporeArmenia El Salvador Madagascar Sint Maarten (Dutch)Australia Eritrea Malawi SlovakiaAustria Ethiopia Malaysia Slovenia
Azerbaijan Fiji Mali SomaliaBahamas Finland Malta South AfricaBahrain France Mauritania South SudanBangladesh Gabon Mauritius SpainBarbados Gambia Mexico Sri Lanka
Belarus Georgia Mongolia State of PalestineBelgium Germany Montenegro SudanBelize Ghana Morocco SurinameBenin Greece Mozambique SwedenBhutan Guatemala Myanmar Switzerland
Bolivarian Republic of Venezuela Guinea Nepal Syrian Arab RepublicBolivia Haiti Netherlands TaiwanBosnia and Herzegovina Honduras New Zealand TajikistanBotswana Hong Kong Nicaragua ThailandBrazil Hungary Niger Togo
Bulgaria Iceland Nigeria Trinidad and TobagoBurkina Faso India North Macedonia TunisiaBurundi Indonesia Norway TurkeyCabo Verde Iran (Islamic Republic) Oman TurkmenistanCambodia Iraq Pakistan Uganda
Cameroon Ireland Panama UkraineCanada Israel Papua New Guinea United Arab EmiratesCentral African Republic Italy Paraguay United Kingdom of Great Britain and Northern IrelandChile Jamaica Peru United Republic of TanzaniaChina Japan Philippines United States of America
Colombia Jordan Poland UruguayCongo Kazakhstan Portugal UzbekistanCongo (Democratic Republic) Kenya Puerto Rico Viet NamCosta Rica Kuwait Qatar Virgin Islands (British)Côte d’Ivoire Kyrgyzstan Republic of Korea Yemen
Croatia Lao People’s Democratic Republic Republic of Moldova ZambiaCuba Latvia Romania Zimbabwe
Academic Policy Committee 5.4: Office of Open Learning Annual Report (2018-2019) Item for: Information
1. Executive Summary A. Introduction The Office of Open Learning (OOL) provides leadership in the design, development and delivery of high-quality online, open, and hybrid learning opportunities for all learners. OOL fosters a teaching and learning environment that provides the tools and policies necessary to empower educators and learners to achieve their aspirations for innovation, excellence, impact, and openness. Underpinned by scholarly practice, we advocate for, explore, and implement the emerging technologies our institution needs to be on the leading edge of educational innovation. The mandate of the Office of Open Learning is to: • Provide support and professional development for instructors of online, open, and hybrid learning and teaching
through formal and informal development opportunities • Use an iterative instructional design process underpinned by scholarship, accepted quality standards, and a
commitment to accessibility and openness in all learning designs • Work collaboratively with faculty and students to design, develop, deliver, evaluate and iterate high-quality,
engaging, media-rich learning environments that respect the diverse disciplinary cultures and contexts of the University’s faculty and students
• Provide rapid eLearning multimedia production services, including emerging techniques such as 360-degree video, immersive learning environments, simulations, augmented and virtual reality, and custom learning objects
• Provide a comfortable, safe and supportive environment in which faculty can explore innovations in technology, pedagogy and curriculum
• Provide faculty and student support for adoption and implementation of emerging educational technologies • Obtain funding for research and development of online and open learning environments • Provide project management of online teaching and learning grants and projects • Assist faculties and departments to design, develop, and increase enrolment in innovative, high-quality online,
hybrid, and open courses and programs • Support faculty in scholarly teaching practice, educational research, and embedding high impact practices in
online, hybrid and open courses • Champion the adoption, adaptation, and development of Open Educational Resources (OERs), including open
access texts, simulations, and other learning tools • Continuously evaluate the needs of online and hybrid learners and educators for highly reliable, usable, and
accessible educational technologies, and collaborate with appropriate campus partners to meet those needs • Expand the boundaries of teaching and learning practice, including emerging educational technologies,
pedagogies, and credentialing practices, and provide evidence-based consultation and advice to faculty on their adoption
• Enhance the University of Windsor and OOL’s profiles as provincial, national, and global leaders in the design and delivery of online education, the Scholarship of Online and Open Teaching and Learning (SoOOTL), and open educational practices
Page 2 of 11
B. Goals and Objectives of Reporting Year Supporting and enhancing quality online, technology-enhanced, and open teaching and learning 1. Continue to partner with individual faculty members, programs, departments, faculties, and central units to
develop successful funding proposals for development and research in online and open teaching and learning 2. Support current eCampus Ontario funded projects successfully through initial launch phases, and assist in any
necessary enhancements to courses once launched and evaluated 3. Continue to partner with faculties, departments, and individual faculty members to expand online and hybrid
course and program offerings, and enrolments to meet SEM Plan and SMA2 targets 4. Champion the adoption, adaptation, and development of Open Educational Resources (OERs), including open
textbooks in courses and programs; begin exploring potential for Z-Cred (zero or low-cost learning resources) courses and programs
5. Continue to develop the Certificate in Online and Open Learning, including the potential to open it up to non-UWindsor affiliates, and exploring options with CEPE for development of programming
6. Partner with Provost’s Office to support a second round pilot of the course redesign framework and Supplemental Instruction
Exploring emerging and innovative technologies to enhance teaching and learning 1. Technology support: Continue Riipen, Labster, and CanCred Open Digital Badges pilots, and begin pilots
with Pressbooks (open publishing tool), Zoom (hybrid video/web conferencing system), H5P (accessible interactive learning objects), Examity online live exam proctoring system (replacing BVirtual), and Mukurtu (indigenous content management system)
2. Continue to support and embed core educational technologies, including Blackboard Collaborate Ultra, and WordPress ePortfolios. Work to transition WordPress to a more suitable host given ongoing growth and develop more specific guidelines for practice/training for students and instructors. Work with campus partners to grow utilisation and embed Lynda.com (now LinkedIn Learning)
3. Continue to explore potential benefits for institutional membership in international organisations such as OERu and the Open Education Consortium
4. Work with broad range of stakeholders including CTL, ITS, Facilities, Classroom Prioritization Committee, student representatives, and others in ongoing work to champion contemporary evidence-based learning space designs that enhance flexibility and support multiple pedagogical choices, provide reliable access to educational technology, and help to facilitate active and technology-enhanced learning
5. Assist UWinsite team in the design and implementation of online training and support tools for the new UWinsite Student system
Evolving and enhancing the OOL to better support the campus community 1. Secure commitment to an appropriate permanent space for OOL that is fit for purpose and serves the campus
community’s needs for educational multimedia, including a rapid eLearning studio/design space and active learning workshop space
2. Launch new OOL website, including a showcase for Open Educational Resources (OERs) created by UWindsor faculty, staff, and students
3. Continue to develop opportunities for meaningful student collaborations and experiential learning to support innovation in online, open, and hybrid learning programs
4. Complete transition of OOL budget to support more strategic needs of the unit and the University community, including securing appropriate staffing and funding for innovative technologies
5. Begin to explore meaningful engagement for the OOL in supporting indigenization of curricula, practices, spaces, and technologies
6. Reintroduce the OOL Strategic Development Grants with a revised focus congruent with institutional and provincial priorities
7. Continue to work on implementation of OOL-CTL joint external review recommendations 8. Enhance the University of Windsor and OOL’s profiles as provincial, national, and global leaders in the design
and delivery of online education, the Scholarship of Online and Open Teaching and Learning (SoOOTL), and open educational practices
Page 3 of 11
C. Successes • Open textbooks: Five open textbooks in development with faculty and student partners; Estimated total
amount saved by students on textbook purchases between summer 2017 to summer 2019 = $540,000 • Continued to expand and support ePortfolio use, with 4,800 users and 4,100 sites launched, 60 workshops
with 1,260 attendees and 110 regular drop-in sessions • Awarded five OER ACE grants worth a total of $31,596 to projects in psychology, education, engineering, and
sociology to support adoption and creation of open educational resources (OERs) • Supported Faculty of Education in design, development and launch of online M.Ed. and intensive
online Technology Education programs and courses (four other new programs in development in other faculties)
• Partnered with nursing, Leddy Library and Western University Health Sciences to complete an environmental scan of OERs for nursing competencies (eCampus Ontario funded) to identify gaps in OER availability; 358 open resources were included in the final scan and mapped against the CNO beginning practice nursing competencies. The resource was used to inform development of two new open textbooks in Ontario, and is now being used to do more work in British Columbia
• Provided 1723 consultations (21% increase over 2017-18) for a total of 2,300 consultation hours (13% increase over 2017-18)
• Certificate of Online and Open Education – 5 graduates in first offering; one course was made available beyond UWindsor and had participants from all over Canada, the US, and South America
• Offered 76 courses, workshops and situated training events (69% increase over 2017-18) • Student Collaborators: OOL projects directly employed 25 student collaborators • 27 local, national and international conference and invited presentations and conference workshops • In partnership with faculty, completed 4 eCampus Ontario funded research and innovation projects
worth $145,283 • Created over 400 video and other custom learning objects • Part of OOL’s mandate is to support the growth of online, open, hybrid and technology-enabled teaching and
learning across the institution. In the reporting year the campus had: o 310 online courses listed in eCampus Ontario portal, including 204 credit and 106 non-credit online and
hybrid courses o 7,488 unique students enrolled in at least one fully online or flexible/hybrid mode course in 2018-19
D. Challenges • Moving office spaces is highly disruptive to work, but the new location will improve working conditions,
especially once renovations are complete • Lack of access to a suitable lab/studio space for efficient and effective rapid eLearning development for
online, hybrid, technology-enhanced, and open access courses places us behind most universities in Ontario; several other institutions are developing ‘one button studios’ for faculty and students to easily record media. OOL is in discussions about renovating some spaces in Lambton Tower basement to serve these functions
• Staffing: Limited capacity in key skill areas (video production, graphic/media design, web-development, 3D media) and reliance on short-term contract hires and student labour continues to be a problem, but steps are being taken to increase capacity
• Access to data on courses and programs remains challenging, especially since the changeover to UWinsite student requiring all previous reports to be rebuilt and very little capacity to do so.
• Ongoing limited technical support for faculty teaching online or hybrid classes out of hours • Limited academic and technical support for online students, including out of hours • Limited range of full programs in online mode makes marketing difficult, although this is changing with new
programs going live • Inconsistent quality and lack of publicly available online information about courses and programs, including
actual delivery mode, and up to date credit transfer data being provided to OnCAT
Page 4 of 11
• Ongoing low capacity of wireless internet access in some parts of the campus, especially for hybrid or technology-enhanced courses and large classes, limits the use of some interactive technologies and potential for secure online exams, and may pose problems for other functions such as online surveys or SETS
• Critical lack of diversity in teaching spaces, especially for active and hybrid teaching and learning • Formal recognition, including in the RTP process, for adopting, adapting, or creating open educational
resources 2. Report A. Area’s Goals and Objectives and the University’s Strategic Plan 1. Provide an exceptional undergraduate experience: a. Continue to partner with individual faculty members, programs, departments, faculties, and central units to
develop successful funding proposals for development and research in online and open teaching and learning a. The incoming Ford government cancelled or postponed most funding opportunities for online and open
learning, so the eCampus Ontario project calls that were expected were cancelled. Going forward, OOL will work with faculty to find suitable alternative funding streams for their ideas.
b. OOL completed the previous round of eCampus Ontario funded projects, including course and program development, OER development, research and innovation, and projects offered for RFP
b. Support current eCampus Ontario funded projects successfully through initial launch phases, and assist in any necessary enhancements to courses once launched and evaluated a. New online courses developed with eCampus Ontario support were successfully launched in Psychology,
Human Kinetics, Physics, Computer Science, and Chemistry, with additional courses to be launched in nursing, psychology, and chemistry over the next 2 years
b. Supplemental resources (question bank, course slides) were created and shared to the community for the OpenStax Astronomy open text book that was adopted in two Astronomy courses; these supplemental resources are now available to the thousands of students and faculty who use the open astronomy texts around the world
c. Previously funded eCampus Ontario and Shared Online Course Fund (ShOCF) projects are being updated and enhanced, including a shared online course (with York and Carleton Universities) transitioning to Windsor, and updating previously funded course development projects
c. Continue to partner with faculties, departments, and individual faculty members to expand online and hybrid course and program offerings, and enrolments to meet SEM Plan and SMA2 targets a. As above, eCampus Ontario funded projects launched multiple new courses (at least 7 launching in 2018-
19). OOL also supported unfunded online course development in chemistry (1), psychology (3), philosophy (3), political science (1), and social work (1). Law also started exploring options for online courses
b. Two new online credit programs launched in Education: M.Ed. (online) and the intensive online and hybrid B.Ed./ Diploma in Technological Studies that was developed and launched in a sprint over summer
c. Supporting the development and delivery of two new non-credit certificate programs through Continuing Education
d. UWindsor currently offers 310 credit (204) and non-credit (106) online courses e. 88% of online enrolments are from Ontario, with 1.3% from other provinces, and 11.5% are international
students; 5,178 students (almost 30% of UWindsor students) enrolled in at least one online course during the reporting period; 7,488 students enrolled in at least one fully online or flexible/hybrid mode courses
d. Champion the adoption, adaptation, and development of Open Educational Resources (OERs), including open
textbooks in courses and programs; begin exploring potential for Z-Cred (zero or low-cost learning resources) courses and programs a. Almost $540,000 saved on textbook costs for students in courses that adopted open textbooks since
summer 2017
Page 5 of 11
b. OOL is supporting the development or adoption of 5 open textbooks and a range of open educational resources
c. Began project to map potential zero textbook cost resources in science, including through open textbook and other OER adoption, and available library resources at no cost to students
d. Provided small OER-ACE grants to 5 projects developing or extending OERs in psychology, education, engineering, and sociology
e. Started transitioning existing OERs to a new, provincially-hosted and more accessible platform (Pressbooks) so they can be more easily discovered
e. Partner with Provost’s Office to support a second-round pilot of the course redesign framework and Supplemental Instruction a. Supplemental instruction/Peer Assisted Learning Sessions (PALS) was expanded into pilots with chemistry
(two large second year courses; multiple sections) and math and statistics (one large cross-disciplinary course; multiple sections)
b. 134 unique students attended PALS sessions across two courses in Winter 2019; overall, students who attended at least one PALS session had a DFW (D-grade, Fail, Withdrawal) rate of 1.9%, compared to 21.5% in the overall course population
c. PALS support was extended with a full-time Limited Term AAS role to be appointed for two years d. Currently exploring rolling out PALS pilots in nursing, economics, the M.Eng. program, and FAHSS
f. Technology support: Continue Riipen, Labster, and CanCred Open Digital Badges pilots, and begin pilots with
Pressbooks (open publishing tool), Zoom (hybrid video/web conferencing system), H5P (accessible interactive learning objects), Examity online live exam proctoring system (replacing BVirtual), and Mukurtu (indigenous content management system) a. Completed several externally funded sandbox technology pilots, including explorations of Riipen micro-
experiences platform, Labster virtual labs and CanCred Digital Microcredential system (ongoing) b. Pressbooks open publishing tool is actively being used for development of open textbooks – UWindsor was
one of the initial pilot group for the provincial open library instance of the system c. Open source H5P interactive learning authoring tool is in pilot phase and integrated with Pressbooks,
WordPress, and some Drupal nodes; exploring options for expanding use d. Mukurtu (open Indigenous content management system) installed and testing e. Zoom web conferencing/Zoom rooms system is being piloted in limited settings; the process of gathering
user requirements for web conferencing systems should be initiated in the coming year f. Successfully migrated from BVirtual to Examity for live online proctoring, including redeveloping resources
for students and instructors. Facilitated 29 exams in 16 courses with 309 student appointments g. Continue to support and embed core educational technologies, including Blackboard Collaborate Ultra, and
WordPress ePortfolios. Work to transition WordPress to a more suitable host given ongoing growth and develop more specific guidelines for practice/training for students and instructors. Work with campus partners to grow utilisation and embed Lynda.com a. Collaborate Ultra usage continues to be high with 17,165 sessions launched from 1 July, 2018-30 June,
2019; 7,712 hours of sessions with 26,681 attendees; 415 hours of recordings were created across 609 recordings; highest number of recordings in a day was 18, maximum attendees in any one session was 134 with the maximum number of attendees using the system at any given time of 474. Blackboard considers UWindsor one of the highest users of Collaborate. Collaborate Classic will be shut down as of 2020 and all users will have to migrate to Collaborate Ultra, with existing recordings from Classic being migrated to a downloadable archive from Amazon Web Services, where Ultra is now housed. There were 74 Collaborate tickets submitted to Team Dynamix and OOL staff provided 184 Collaborate consultations during the reporting period
b. OOL staff facilitated 60 ePortfolio workshops with approximately 1,260 students attending; 110 ePortfolio drop-in sessions, and 51 ePortfolio consultations with students, staff, and faculty. There were 774 new users and 759 new ePortfolios created during the reporting period. There are currently 4,124 WordPress ePortfolio sites in the system, with 4,780 users signed up.
Page 6 of 11
c. Lynda.com, which was licenced for all post secondary education students and employees by the Province on a 3-yr trial was purchased by Microsoft and became LinkedIn Learning. This requires a new authentication protocol and other adjustments by ITS, which have to be completed by Aug. 1, 2019 to maintain UWindsor access to the tool. All access to previous Lynda.com links will be removed as of March 31, 2020. During the reporting period, UWindsor users watched a total of 84,745 videos (average 22.7 per user) for a total of 5,601 hours; user accounts doubled to 2,224 with an average of 1,384 logins per month.
h. Work with broad range of stakeholders including CTL, ITS, Facilities, Classroom Prioritization Committee,
student representatives, and others in ongoing work to champion contemporary evidence-based learning space designs that enhance flexibility and support multiple pedagogical choices, provide reliable access to educational technology, and help to facilitate active and technology-enhanced learning a. This project is ongoing. Initial meetings have been held with several stakeholders and data provided to the
committee. OOL and CTL have committed to developing instruments for capturing post-occupancy data on renovated classroom spaces to inform future designs.
i. Assist UWinsite team in the design and implementation of online training and support tools for the new UWinsite Student system a. The UWinsite team scripted six videos that OOL edited and produced; these videos were viewed over 2,600
times during the reporting period
j. Secure commitment to an appropriate permanent space for OOL that is fit for purpose and serves the campus community’s needs for educational multimedia, including a rapid eLearning studio/design space and active learning workshop space a. OOL, CTL and Quality Assurance began the process of collocating in a recently vacated space in the Centre
for Engineering Innovation, which brings together teaching and learning support services into a single space on campus for the first time. The space requires relatively extensive renovations to be usable and meet the needs of the three units, but once complete this move provides enough space for current staff of the three units, plus spaces to support core functions such as workshops and meetings. Renovations, including installation of a phone, wired and wireless network, AV and creation of new offices are ongoing.
b. OOL is continuing to collaborate with PAC to identify and develop a multi-purpose space for a creating educational multimedia. This process is ongoing.
k. Continue to develop opportunities for meaningful student collaborations and experiential learning to support innovation in online, open, and hybrid learning programs a. Students are involved in all OOL development projects, and students are employed in a variety of roles
including as PALS leaders, Outstanding Scholars, interns, media assistants, and project assistants. Twenty-five students worked directly with OOL in the reporting period
l. Begin to explore meaningful engagement for the OOL in supporting indigenization of curricula, practices, spaces, and technologies a. The OOL facilitated a community of practice over a semester that read and explored the UBC open
textbook, A guide for Indigenization of post-secondary institutions. b. OOL grants specifically encourage applications that address Indigenous needs and Indigenisation of the
curriculum c. OOL is currently testing the capabilities of the Mukurtu open Indigenous content management system for
potential use in Indigenous knowledge projects d. The OOL team have committed to incorporating Indigenous knowledge, traditions, and authors in our own
teaching activities
Page 7 of 11
2. Pursue strengths in research and graduate education:
a. Enhance the University of Windsor and OOL’s profiles as provincial, national, and global leaders in the design and delivery of online education, the Scholarship of Online and Open Teaching and Learning (SoOOTL), and open educational practices a. Completed 4 research and innovation projects funded by eCampus Ontario; results were presented at
multiple international, national and local conferences, and the results of one study were published in a peer reviewed journal, with more publications in process
b. Completed a contract environmental scan ($34,440) of open educational resources in nursing in collaboration with Leddy Library, Nursing, and Western University. The project identified hundreds of OER resources mapped to the College of Nurses Ontario competencies. The project was openly licenced and has since been use to inform development of significant OERs in Ontario and has recently been picked up by BC Campus as the basis for developing new OERs in nursing in British Columbia.
c. OOL supports the scholarship of online and open teaching and learning in faculty and students involved in our projects, and often co-presents or co-publishes with our collaborators. In 2018-19, OOL faculty presented or co-presented 18 conference papers at local, provincial, national, and international conferences, 9 invited or keynote presentations, and published one peer-reviewed journal article. The first cohort of COOL graduates is also working to publish a paper on their experiences in the program.
d. OOL staff also serve on organising committees for local, provincial, national and international conferences e. Support Three Minute Thesis: OOL provides training and support for UWindsor 3MT contestants in
collaboration with Grad Studies and the GATA network
b. Continue to develop regional events on online, technology enhanced and open teaching and learning in partnership with local institutions a. Following the success of the first Open Education Summit, which was launched as a partnership between
UWindsor, St Clair College, and Lambton College in May, 2018, it was decided that OOL would host a regional conference focused on technology in teaching, online and open education every second year, with the next event planned for summer 2020
b. OOL staff also participated in regional teaching and learning events at St. Clair College and for local K-12 teachers
3. Recruit and retain the best faculty and staff: a. Continue to develop the Certificate in Online and Open Learning, including the potential to open it up to non-
UWindsor affiliates, and exploring options with CEPE for development of programming a. Enrollment remains stable in the core COOL courses, with 34 registrants in 2018-19. b. Five faculty and graduate students completed the COOL program in its inaugural; 3-5 graduates are
expected in the second cohort to graduate in November 2019. The first cohort has presented papers on their experience in the program at local and national conferences, and co-authored a paper (expected publication in 2020)
c. A review of the program after it’s first year led to some streamlining and clarification of program requirements. Three alternative program designs were developed for learners in different contexts: UWindsor affiliates (free), continuing education participants (cost recovery), and open courses (free and openly licenced). These will continue to be developed in the coming year.
d. Several other universities and colleges have expressed interest in using the COOL program locally with their faculty and one institution has inquired about licencing the materials
e. The program was featured in Academica, which led to one of the courses being experimentally opened up to participants from across Canada, the US and South America; as a result of this, options for offering different versions to different audiences are being explored, with an initial market analysis done by Continuing Education.
f. OOL also facilitated a Community of Practice on online, open and technology enabled learning that met 6 times throughout the year (approximately 30 attendees, most came to more than one session)
Page 8 of 11
b. Complete transition of OOL budget to support more strategic needs of the unit and the University community, including securing appropriate staffing and funding for innovative technologies a. Budget transition will be complete in 2020-21 budget year to a new model that better supports the needs
of the unit and institution, while gradually phasing out historical elements b. The new model includes increased staffing to support course and resource development, media
production, re-introduction of small grants, supporting Supplemental Instruction/PALS pilots, online program development, new technologies, and other special projects
c. Reintroduce the OOL Strategic Development Grants with a revised focus congruent with institutional and
provincial priorities a. Grants were reintroduced in 2019 with a theme of adopting, creating and extending open educational
resources b. Five grants were awarded to projects in psychology, education, engineering, and sociology (Total awarded:
$31,596). Projects expected to be completed by spring 2020. d. Continue to work on implementation of OOL-CTL joint external review recommendations
a. Work is ongoing to implement the recommendations within our control; Co-location of OOL with CTL in the CEI achieves one of the main recommendations. OOL and CTL have been working to develop shared understanding of roles and to streamline workflows for areas of crossover, collaboration and synergy. We have also reviewed our mission, mandate and vision for potential efficiencies. The three units moving to the CEI will host a formal open house and information campaign to introduce the campus to the new space and services once renovations are complete.
4. Engage and build the Windsor and Essex County community through partnerships: a. Launch new OOL website, including a showcase for Open Educational Resources (OERs) created by UWindsor
faculty, staff, and students a. New website contains an expanded library of resources to support faculty and students in online, open and
hybrid learning and teaching, and to showcase projects, funding, and open educational resources to the broader community. Site is awaiting accessibility audit so it can be launched
b. Champion the adoption, adaptation, and development of Open Educational Resources (OERs), including open
textbooks in courses and programs; begin exploring potential for Z-Cred (zero or low-cost learning resources) courses and programs a. See 1 d. above. Also supporting student-led development of OERs that can be shared with the local
community and beyond, including resources built for teachers to inform practice in educational technology (partnership with Education)
c. Begin to explore meaningful engagement for the OOL in supporting indigenization of curricula, practices,
spaces, and technologies a. See 1. l above. Additionally, OOL is exploring opportunities for partnerships with local Indigenous groups
such as the Can-Am Indian Friendship Centre, and Walpole Island Heritage Centre, particularly around educational technologies and protection of Indigenous knowledge/Intellectual Property online
d. Continue to develop regional events on online, technology enhanced and open teaching and learning in
partnership with local institutions a. See 2a) above. b. Additionally, OOL is continuing to work with departments and St Clair College to develop online transfer
pathways, and has begun discussions with CEPE to identify possible opportunities to support online programming for continuing education
c. OOL collaborated with St Clair College to deliver workshops to faculty in programs where transfer credit arrangements exist or are being considered
Page 9 of 11
5. Promote international engagement:
a. Continue to explore potential benefits for institutional membership in international organisations such as OERu and the Open Education Consortium a. UWindsor is an institutional member of the International Consortium of Distance Educators (ICDE) and this
year joined the Open Education Global (OEGlobal) Consortium (memberships funded by the OOL) b. OOL faculty and staff continue to be active in scholarship, often partnering with faculty and grad students
to present their scholarship at local, national, and international conferences with 27 conference and invited presentations in 2018-19;
c. OOL’s Twitter presence: 88,513 impressions in 2018-19 (average 7,376/mth), 866 followers
B. Future Actions/Initiatives Supporting and enhancing quality online, technology-enhanced, and open teaching and learning 1. Champion the use of Open Educational Practices (OEPs) including creation and use of Open Educational
Resources (OERs) and zero cost resources and in all teaching contexts 2. Map availability of OERs for major programs on campus and make those available to support adoption in
courses and programs 3. Expand Supplemental Instruction (PALS) into new disciplines and courses, and at the graduate level 4. Partner with stakeholders across campus to develop definitions of course delivery modes congruent with
provincial and national reporting requirements and support change management for course data collection workflows
5. Continue to partner with faculties, departments, and individual faculty members to expand open, online and hybrid offerings and enrolments to meet the SEM Plan and SMA3 targets
6. Continue to evolve the Certificate in Online and Open Learning (COOL) to serve multiple audiences 7. Contribute to the evolution of teaching practice as members of the Student Evaluation of Teaching (SET)
Taskforce Exploring emerging and innovative technologies to enhance teaching and learning 1. Continue to expand ePortfolios for all students, including upgrading infrastructure and expanding support, to
support SMA3 targets 2. Explore opportunities for microcredential pilots, and work with stakeholders to develop a framework for
identifying and offering microcredentials systematically in credit and non-credit contexts 3. Explore opportunities to pilot a Domain of One’s Own project 4. Continue pilots with Pressbooks (open publishing tool), Zoom (hybrid video/web conferencing system), H5P
(accessible interactive learning objects, and Mukurtu (indigenous content management system) 5. Work with eCampus Ontario to pilot an open LMS 6. Explore options for a robust online assessment tools ecosystem, including transitioning Examity live online
exam proctoring system to a new contract, and gather requirements for other services that may be needed (e.g. digital exam creation and delivery tools, assessment distribution infrastructure, lockdown browser tools)
7. Support the rollout of Equatio and Write accessibility tools in collaboration with OHREA and Student Services 8. In collaboration with CTL, pilot Mobius in Mathematics and Statistics 9. Explore options for video asset management and personal video capture tools 10. Continue to support core educational technologies, including Blackboard Collaborate Ultra, and
WordPress ePortfolios 11. Work with broad range of stakeholders including CTL, ITS, Facilities, Classroom Prioritization Committee,
student representatives, and others in ongoing work to champion contemporary evidence-based learning space designs that enhance flexibility and support multiple pedagogical choices, provide reliable access to educational technology, and help to facilitate active and technology-enhanced learning
Page 10 of 11
Evolving and enhancing the OOL to better support the campus community 1. Complete move and renovations to new space in the Centre for Engineering Innovation 2. Secure commitment to an appropriate permanent space that is fit for purpose and serves the campus
community’s needs for educational multimedia, including a rapid eLearning studio/design space and active learning workshop space, sound recording and podcasting spaces
3. Update and offer small internal grants to support online, technology-related and open projects with a focus congruent with institutional and provincial priorities, including contributing to SMA3 and enrollment targets
4. Increase staffing capacity to support campus needs for OOL services in key skill areas 5. Develop a new conference to run every second year that is focused on online, open, and technology-enhanced
teaching and learning issues 6. Launch new OOL website, including a showcase for Open Educational Resources (OERs) created by UWindsor
faculty, staff, and students 7. Continue to explore meaningful engagement in indigenization and decolonization of curricula, practices,
spaces, and technologies 8. Continue to develop opportunities for meaningful student collaborations and experiential learning to support
innovation in online, open, and hybrid learning programs 9. Continue to raise the University of Windsor and OOL’s profiles as provincial, national, and global leaders in the
design and delivery of online education, the Scholarship of Online and Open Teaching and Learning (SoOOTL), and open educational practices
C. Recommendations for Senate consideration 1. Recommend APC or a sub-committee to develop a transparent set of definitions of all characteristics of
courses and programs (e.g. on-campus, online, hybrid, flexible, technology enhanced, intensive, active learning, zero or low cost resources) that aligns with provincial and national reporting requirements, make these characteristics visible to students and the public, and adjust workflows to collect this information reliably and accurately. Rationale: a. It is currently very difficult for students to find any accurate and timely information about the characteristics
of courses, including their delivery mode, assessment approaches, and required learning resources, and for students from outside UWIndsor, it is impossible because there is no public-facing course catalogue. This places UWindsor at a significant disadvantage compared to other institutions who make such information available as a recruitment tool
b. It is currently impossible to accurately and efficiently report or provide data to provincial and national bodies on course and section attributes such as delivery mode because this data is not reliably collected through current workflows. Providing institutional definitions and the ability to describe course characteristics, such as delivery mode, experiential learning, no-cost learning resources etc., effective education on these definitions, and an effective workflow for capturing this data will improve the quality of the data, and allow it to be displayed to students and other users
2. Begin a process to define and recognize microcredentials as part of the institutional credential ecosystem.
Strike a sub-committee or taskforce to investigate microcredentials and develop a framework for these. Rationale: a. The University does not currently have a policy or guidelines on microcredentials, and given the current
focus on these from the provincial government, and in the SMA3 agreement, it is important that there is an institutional understanding of microcredentials. Consideration should be given to questions such as what they are, what data should be included in the digital credential, what level of evidence is expected, guidance on laddering of microcredentials to higher credentials, the relationship to other credentials, potential benefits to students, and how and where they may be awarded
Page 11 of 11
3. Consider ways to recognize and reward engaging with Open Educational Practices (including adapting, adopting or creating open textbooks and other Open Educational Resources) and Open Knowledge Creation through the RTP process. Explicit acknowledgement of open practices in RTP forms and guides as evidence for departments to consider in developing their own RTP criteria for teaching and research would be a low-stakes means of encouraging this practice. Rationale: a. There is strong evidence for both the cost-saving and enhanced student learning outcomes of Open
Educational Practices (OEPs), and the transformative pedagogical and social potential of engaging in such practices. However, faculty often find investing time in developing, adopting, and adapting OERs to be challenging because it is not recognized explicitly in the RTP criteria of the institution. There has been a strong movement in the US and elsewhere to develop mechanisms to reward and recognize engagement with OEPs, and some Canadian institutions are also now supporting faculty in this way (KPU, SFU and UBC all recognize it in RTP). Institutions that engage meaningfully with OEPs and utilize OERs and other low or no-cost learning resources are seeing significant enrollment increases and retention improvements. UWindsor already has a memo endorsed by Senate to encourage faculty to consider OERs in selecting learning resources, but they need recognition through the processes that matter most to faculty, including RTP.
b. Considering the requirements for open publishing of research data from Tri-council and many other funding agencies, explicitly recognizing the practice of open knowledge dissemination including through open publishing, as part of RTP research criteria is important in normalizing this practice
