Overview and Discussion of NRS Changes
Massachusetts Department of Elementary and Secondary Education
2
NRS Changes for FY13
AGENDA Review changes on follow-up outcomes
and new data requirements Review criteria for cohorts Review changes to reporting timeline
and reporting tables Review questions and answers posted on
the ACLS website Respond to participant questions
Massachusetts Department of Elementary and Secondary Education
3
Time Period Definitions Program Year (PY)– Twelve month period
from July 1 through June 30 defined by the calendar year at start of period. (federal terminology) For example, July 1, 2012 through June 30,
2013 is the time period corresponding to PY12. Fiscal Year (FY)– Twelve month period from
July 1 through June 30 defined by the calendar year at end of period. (state terminology) For example, July 1, 2012 through June 30,
2013 is the time period corresponding to FY13.
Massachusetts Department of Elementary and Secondary Education
4
Changes to Follow-up Outcomes: Automatic Cohort
DesignationProgram Year (FY2012): Participants identified based on self-reported goals (program – level)
Program Year (FY2013): Participants automatically identified based on “cohort” which are based on demographic variables in the MIS system (state – level)
The definition for “cohort” varies by outcome.
Massachusetts Department of Elementary and Secondary Education
5
Implications of “Automatic Cohort Designation”
Students belong to a group to be followed for NRS post-program outcomes based on pre-defined characteristics. Following students for post program outcomes willno longer be goal dependent. As has always been the case, we are only
concerned with students having at least 12 hours of attendance.
Massachusetts Department of Elementary and Secondary Education
6
YES!!
Local staff should continue to set goals with students
Goal setting is an essential component of a good
program and good instruction but….
Students goals will no longer be used as a way to
identify students for the NRS follow-up measures
Do Student Goals Still Matter?
Massachusetts Department of Elementary and Secondary Education
7
Massachusetts Department of Elementary and Secondary Education
8
Secondary Credential Cohort
Obtain Secondary Credential Cohort Learners who take all GED tests, or Are enrolled in adult high school at the high
ASE level, or In the assessment phase of the EDP, and Who exit during the program year
Entry into cohort occurs only for those who take all 5 GED tests by the end of reporting period (December 31).
Massachusetts Department of Elementary and Secondary Education
9
Q: Do students have to still be enrolled when they take the tests?
No, students are included in the cohort if they take all five test before the end of the reporting period for the program year in which they exit (December 31)
Q: Must a student pass all GED tests to be included as achieving the outcome?
Yes, students must pass all GED tests to be included as achieving the outcome
Question: Cohort for Exited Students and GED Tests
Massachusetts Department of Elementary and Secondary Education
10
Example 1: A student who enrolls in fall of 2012, exits in March 2013, takes 3 tests then, takes 2 more in September 2013, would be included in the cohort for 2012-13
Example 2: A student who enrolls in October 2012, takes 2 GED tests in May 2013, exits in November 2013 and takes 3 GED tests would be included in the cohort for 2013-14
Example 3: A student who enrolls in December 2012, exits in March 2013, takes 3 tests then, takes 2 more in January 2014, would not be included in any cohort
Passing GED Tests: Examples
Massachusetts Department of Elementary and Secondary Education
11
Postsecondary Cohort
Postsecondary CohortAll learners who
have earned a secondary credential while enrolled
hold a secondary credential or are enrolled in a class specifically
designed for transitioning to community college
and who exit during the program year
Massachusetts Department of Elementary and Secondary Education
12
Questions: Foreign Credential and Transition class
Q: Are students with a secondary credential, earned in another country, in this cohort? Are ESOL students to be included?
Yes, these students are included in the cohort
Q: How is a transition class defined ? A class that has a specific purpose to prepare students
for entry into postsecondary education, training or an apprenticeship program.
The rate based class focus would be Bridge to College/Math or Bridge to College/Writing. The non rate based class focus would be Bridge to College/Math, Bridge to College/Writing, or College and Career Readiness.
Massachusetts Department of Elementary and Secondary Education
13
Q: Please clarify multi-year reporting of entry into postsecondary education
Students who exit during program year achieve this outcome if they enter postsecondary any time from exit through the end of the following program year (June 30)
Follow students and report for two years, in two rows on Table 5 (unduplicated) First report (2012-13): report same as of now Future years: Update prior year—keep cohorts
separate First two year reporting on 2013 report (due December 31, 2014)
Reporting of Entry into Postsecondary--Timing
Massachusetts Department of Elementary and Secondary Education
14
Entered Employment Cohort Learners who are not employed at time of
entry and in the labor force who exit during the program year
Employment Status in SMARTT: Employed Unemployed and Looking for Work Unemployed and Not Looking for Work Retired or Otherwise Not Looking for Work Homemaker Programs should exit students who are no longer in attendance by filling out the “Exit Student” screen in SMARTT.
Employment Cohorts
Massachusetts Department of Elementary and Secondary Education
15
Retained Employment Cohort Learners who were not employed at time of
entry and in the labor force, who are employed by the first quarter after exit quarter
Learners employed at entry who exit during the program year
Employment Status in SMARTT: Employed Unemployed and Looking for Work Unemployed and Not Looking for Work Retired or Otherwise Not Looking for Work HomemakerPrograms should exit students who are no longer in attendance by filling out the “Exit Student” screen in SMARTT.
Employment Cohorts
Massachusetts Department of Elementary and Secondary Education
16
Students, in entered employment cohort, that are employed the first quarter after exit are also in the retained employment cohort
The timeframe is the same as always–it has not been changed
States must use students’ employment status at entry to determine cohort status for employment outcomes.
Reminder on Retained Employment
Massachusetts Department of Elementary and Secondary Education
17
Q: How is “unemployed, in the labor force defined?”
NRS definition (Unchanged): Learners who are not employed but are seeking employment, make specific efforts to find a job and are available for work
Employment Status in SMARTT: Unemployed and Looking for Work
Question: Definition of Unemployed in Labor Force
Massachusetts Department of Elementary and Secondary Education
18
Entered EmploymentEmployed in the first quarter after exit.
Retained EmploymentEmployed in the third quarter after exit.
Timing of Employment Measure
Massachusetts Department of Elementary and Secondary Education
19
Collection: One quarter after exit for entered employment, three quarters after exit for retained employment
Reporting lag (previous year) to accommodate lag in Unemployment Insurance (UI) wage records
Review: Employment Data Collection and Reporting
Massachusetts Department of Elementary and Secondary Education
20
Exit Quarter MeasureCollect
Report
July 1–September 30, 2012(First Quarter)
Entered Employment October 1–December 31, 2012(Second Quarter FY 2013) FY 2013
Retained Employment April 1–June 30, 2013(Fourth Quarter FY 2013) FY 2013
October 1–December 31, 2012
(Second Quarter)
Entered Employment January 1–March 31, 2013(Third Quarter FY 2013) FY 2013
Retained Employment July 1–September 30, 2013(First Quarter FY 2014) FY 2014
January 1–March 31, 2013(Third Quarter)
Entered Employment April 1–June 30, 2013(Fourth Quarter FY 2013) FY 2013
Retained Employment October 1–December 31, 2013(Second Quarter FY 2014) FY 2013
April 1–June 30, 2013(Fourth Quarter)
Entered Employment July 1–September 30, 2013(First Quarter FY 2014) FY 2014
Retained Employment January 1–March 31, 2014(Third Quarter FY 2014) FY 2014
Massachusetts Department of Elementary and Secondary Education
21
Enter Employment/Get a Job -- Those with SSN and signed release form. For all others, survey method must be used.
Retain Employment -- Those with SSNand signed release form. For all others, survey method must be used.
Survey form may be accessed at:http://www.doe.mass.edu/acls/smartt/
nrssurvey_followup.pdf
Data Matching Department of Revenue
Massachusetts Department of Elementary and Secondary Education
22
Obtain GED -- All students (regardless if SSN provided and release form signed since match will be with the Oklahoma Scoring Service)
Enter Postsecondary Education -- -- All students (regardless if SSN provided and release form signed since match will be with the National Student Clearinghouse)
Data Matching Oklahoma Scoring Service, and National Student Clearinghouse
Massachusetts Department of Elementary and Secondary Education
23
Follow-up Outcome Timing Secondary CredentialObtained credential before the end of the reporting period (December 31)
Postsecondary EducationEntered any time from exit until the end of the following program year (June 30)
Report for two years, in two rows on Table 5 (unduplicated)Programs should exit students who are no longer in attendance by filling out the screen found by clicking on the “Exit Student” menu option within SMARTT .
Massachusetts Department of Elementary and Secondary Education
24
Reporting Changes: Table 5 Table 5 revised to include instructions for automatic cohort designation
New row added for postsecondary transition: Report two years (unduplicated)
Update prior year First two year reporting on 2013 report
(Due December 31, 2014)
Massachusetts Department of Elementary and Secondary Education
25
Reporting Postsecondary Example in 2013-14
In 2013-14, 125 students in cohort exit, state
matches 100 and finds 50 have entered postsecondary
State runs match on 2012-13 cohort again by
June 30, 2014 and finds another 20 students have entered postsecondary in addition to the 40 from previous year
Massachusetts Department of Elementary and Secondary Education
26
Example of Table 5: 2013-14
Core Follow-up Outcome Measures
Number of Participants in
Cohort
Number of Participants Included in
Survey Sample
Number of Participants Responding to Survey or
Used for Data
Matching
Response Rate or Percent
Available for Match
Number of Participants Achieving Outcome
Percent Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)Entered Postsecondary Education or Training -- current program year
120 100 83%
(100/120)
40 40%
(40/100)
Entered Postsecondary Education or Training – prior program year
Leave entire row blank
Massachusetts Department of Elementary and Secondary Education
27
Example of Table 5: 2014-15
Core Follow-up Outcome Measures
Number of Participants in
Cohort
Number of Participants Included in
Survey Sample
Number of Participants Responding to Survey or
Used for Data
Matching
Response Rate or Percent
Available for Match
Number of Participants Achieving Outcome
Percent Achieving Outcome
(A) (B) (C) (D) (E) (F) (G)Entered Postsecondary Education or Training -- current program year
125 100 80% 50 50%
Entered Postsecondary Education or Training – prior program year
120 100 83% 60
(prior year was 40,
now 40+20)
60%
(60/100)
Massachusetts Department of Elementary and Secondary Education
28
Implications for Implementation The number of learners to follow will
increase.
Obtaining SSN’s and signed release forms will lessen the follow-up burden.
Local staff will need to follow-up by survey method for cases where SSN’s and signed release forms were not obtained for those within any of the employment cohorts.
Massachusetts Department of Elementary and Secondary Education
29
Q: How should we determine years of student education? Do we need to check? Self-reported by student, NRS does not
require verification but the state may choose to do so
Q: How do we count education in other countries that is different than our system? Use years of schooling Use in-country definition for secondary
credential
Questions: Reporting Student Education
Massachusetts Department of Elementary and Secondary Education
30
Q: Can students belong to multiple cohorts? Yes, for example, an unemployed student in
the labor force, with secondary credential who exits, would be in two cohorts
Question: Multiple Cohorts
Massachusetts Department of Elementary and Secondary Education
31
Q: Do we have to re-identify cohort membership for students continuing the next year? Yes, for example, employment status or
secondary credential status may change But only collect measure at exit
Question: Continuing Students
Massachusetts Department of Elementary and Secondary Education
32
ADDITIONAL QUESTIONS??