1

Missing Link Project Update

TAIR Annual Conference

February 6, 2008

James LoiselleHiwot Berhane

Texas Workforce Commission- Missing Link Project

2

• Background

• ICF Approach

• Overview of Results

• Limitations

• Questions/Comments

Agenda

3

Job creation and economic development a statewide priority

• Industry clusters facilitate this by:

• Helping Texas efficiently monitor workforce conditions

• Bringing attention to industries and jobs key to the State’s growth

Purpose of project:

• Develop master database linking DWAs, CIPs, and WECM courses

• Provide a common set of linkages to bridge the gap between employers and workforce trainers

Background

4

Aligning WECMs, CIPs, and DWAs – A Stepwise Approach(8 Total Tasks)

Step 2: Edit and verify DWA to CIP links made by TWC/LMCI staff

Step 3: Link DWA statements to WECM courses

Step 4: Gather SME Input to Refine and Validate Ratings

Step 1: Project Kick-off

5

• Gathered information regarding CIP, DWA, and WECM data sources

• TWC provided database of 411 ‘technical’ CIP codes with unverified linkages to DWAs and SOC codes.

Task 1 – Project Kick-Off

6

Database Structure of ‘Mechanical’ Linkages Provided By TWC

■ CIP Code Related O*Net Occupations

DWA 1

■ DWA 2

■ DWA 3

■ DWA 4

7

• ICF reviewed linkages made by TWC staff between the O*NET occupations and CIP codes

• Remaining non-technical CIPs were removed from the original database

• Outcome: Database of 384 ‘technical’ CIP codes with unverified linkages to DWAs and SOC codes.

Task 2 – Edit And Verify CIP and DWA Database Structure

8

• Decided: Linking DWA to WECMs → best way to demonstrate a link between a DWA and a CIP

• Developed one Access database for each of the 384 CIPs

• Created a specialized Access ‘form’ within each database to aid the rating process

• Conducted ratings of each DWA to WECM relationship for each CIP

• Outcome: Complete ratings for all potential DWA to WECM relationships in every CIP-specific database

Task 3 – Link DWA statements to WECM courses

9

Number of Elements in Rating Process and Estimated Ratings Made By ICF Team

Total Number of CIPs 384

Number of CIPs with related WECM information 309

Number of ‘Blank’ CIPs (i.e., no related WECMs) 75

Number of WECMs included in linkage process* 4334

Number of DWAs included in linkage process** 2165

Approximate number of DWA to WECM linkage ratings 350,000

Task 3 – Link DWA statements to WECM courses

*Note 1: 4461 WECMs were provided by TWC in original database. After 127 duplicates were removed, 4334 WECMs remained.

**Note 2: Individual DWAs, by nature, were typically linked to multiple CIPs.

10

Task 3 – Link DWA statements to WECM courses

■ CIP Code Occupations

■ Course 1 DWA 1

■ Course 2 DWA 2

■ Course 3 DWA 3

■ Course 4 DWA 4

11

Task 3 – Link DWA statements to WECM courses

Scale to Determine DWA to WECM linkage

Is a student better prepared to perform the DWA after completing this WECM?

Rating Option Description Yes DWA covered in the WECM.

No DWA not covered in the WECM.

Maybe: Unclear CIP/WECM

WECM includes technical term requiring input from Subject Matter Experts (SMEs).

Maybe: Unclear DWA DWA includes technical term requiring input from SMEs.

Maybe: Don't Know DWA and WECM content are clear, but need SME input to make judgment.

Special Topic WECM is a “Special Topics” Course: content subject to change over time and across schools. Unable to identify reliable linkage.

12

Summary of Rating Rules Used to Distinguish Yes/No Ratings

Rating SituationYes Student is better prepared to perform the DWA after

completing WECM than before the WECM. Yes WECM introduced basic elements of DWA although

student does not have complete mastery of the DWA.Yes WECM introduces the theory behind a DWA although

student has not performed the DWA in the WECM.No DWA involves providing general supervision or performing

a broad activity not related to the content of the WECM .No Individual who has just completed WECM is no better at

performing DWA than prior to WECM.No Student may be better able to perform DWA after WECM

but performance improvement is not related to WECM content

• Intent was to identify WECMs where knowledge or skill related to DWA could be obtained.

• Distinguishing Yes/No ratings sometimes difficult so rules developed for raters/ SMEs:

Task 3 – Link DWA statements to WECM courses

13

• Gathered input from SMEs to refine/validate the DWA to WECM ratings

• Assigned databases based on SME area of specialization.

• SMEs reviewed three types of ‘Maybe’ ratings to change the ‘Maybe’ ratings into Yes/No ratings.

• SMEs reviewed existing Yes/No ratings.

• SME selection, use, and affiliation described on subsequent slides.

• Outcome: Verified ‘Yes’, ‘No’, or ‘Special Topic’ rating for each DWA to WECM relationship.

Task 4 – Gather SME Input to Refine and Validate Ratings

14

Task 4 – Gather SME Input to Refine and Validate Ratings

Selection and Use of Project SMEs:

• SMEs selected based on knowledge of technical CIPs under study.

• CIPs clustered into 28 ‘Program Areas’ (e.g., Mathematics and Statistics, Visual and Performing Arts, Engineering, Construction Trades).

• SMEs asked to indicate ‘areas’ of expertise using defined program clusters.

• SMEs used to resolve ‘Maybes’ where raters had questions about CIP/ WECM content and/or technical aspects of work being performed.

• 32 SMEs were used to rectify ‘Maybe’ ratings across all CIPs and verify Yes/No ratings

• SMEs included technical college faculty, university grad students, and education/work specialists from 10 organizations including ICF.

15

Summary of Participating SMEs

Number of SMEs

Organization Location

1 Temple College Temple, TX3 Angelina College Lufkin, TX1 San Jacinto College Pasadena, TX1 Dallas County Community College Dallas, TX1 Lone Star Cy-Fair College Cypress, TX3 Texas A&M University College Station, TX5 Angelo State University San Angelo, TX1 Penn State University State College, PA8 George Mason University Fairfax, VA8 ICF International Fairfax, VA

32 SMEs 10 Organizations

Task 4 – Gather SME Input to Refine and Validate Ratings

16

Step 6: Conduct roll-up for DWA to CIP linkages

Step 7: Link DWA statements to WECM courses

Step 8: Create master database

Step 5: Identify and Add Missing DWAs for Each CIP

Aligning WECMs, CIPs, and DWAs – A Stepwise Approach(8 Total Tasks)

17

• Reviewed each CIP database to determine whether additional DWAs should be added

• Compared the original list of linked DWAs to the full list of DWAs provided in O*NET, which were clustered into program (CIP) groups

• Outcome: All relevant DWAs represented in CIP to DWA linkages.

Task 5 – Identify and Add Missing DWAs for Each CIP

18

Task 5 – Identify and Add Missing DWAs for Each CIP

■ CIP Code Related O*Net Occupations

■ Course 1 DWA 1

■ Course 2 DWA 2

■ Course 3 DWA 3

■ Course 4 DWA 4

DWA 18DWA 18=Newly identified DWA currently addressed through WECM but not originally linked to CIP through ‘mechanical’ linkages conducted by TWC.

19

• Used roll-up strategy to define relationships between DWAs and CIPs.

• DWAs considered linked to a CIP (i.e., assigned a ‘Yes’ rating) if linked to one or more WECMs related to the CIP.

• CIPs considered not linked (i.e., assigned a ‘No’ rating) if not linked to any WECMs related to the CIP

• Outcome: ‘Yes’/ ‘No’ rating indicating DWAs linked to each CIP

Task 6 – Conduct Roll-Up for DWA to CIP Linkages

20

• Reviewed each CIP database to verify all ratings accurate, structure and content of the databases sound, and redundancies removed.

• Removed duplicate WECMs, verified variable/field name consistency, and updated forms to facilitate data read, edit, and present functions.

• Outcome: Final databases for each CIP.

Task 7 – Conduct Quality Control Check of Databases

21

• Created a user-friendly, searchable master database that incorporates and reflects the new and edited relationships between WECM Courses, CIP codes, and DWAs.

• Outcome: Final master database combining all linkages across CIPs, DWAs and WECMs while providing easy search capability.

Task 8 – Create Database of Identified Linkages

22

Results- Deliverable Overview

Exhibit III-2 Deliverable Overview

Deliverable Category 1 Deliverable Category 2 Deliverable Category 3 309 CIP databases (MS

Access 2003) Each CIP database

contains: 3 tables 1-3 queries 2 forms

Purpose of databases is to

archive detailed data at the CIP-DWA-WECM level along with the queries and forms used at various stages of the project.

1 consolidated database (MS Access 2003)

Consolidated database

contains: 309 CIP tables and 1

WECM table Purpose of database is to

provide data administrator(s)/analysts with one comprehensive source containing all CIP information and corresponding DWA-CIP relationships.

75 blank CIP databases (MS Access 2003)

Each blank CIP database

contains: 2 tables 1 query

Purpose of database is to

provide TWC with CIP databases that do not have any WECMs associated to the CIP. Having 0 WECMs linked, this CIP tables have 0 rating columns, hence the name blank database.

23

• 309 CIP Databases with 3 tables and 2 forms each. Sample shown below:

Tables FormsCIP150305 CIP150305

CIP150305_RollUpWECM_w_CIPinfo_listofsocs_Master

WECM_w_CIPinfo_listofsocs_Master

Deliverable Category 1

24

• Consolidated Database with 309 tables each representing one CIP.

Deliverable Category 2

25

• Select Forms on Objects menu > Open WECM_w_CIPinfo_listofsocs_Master

• View CIP details, related WECMs with details, and related DWAs with ratings

Navigating a CIP Database - 521904

26

• O*NET DWAs are sometimes vague, incomplete, and/or not aligned with the needs of Texas industries limiting the application of results

• WECMs do not cover the gamut of skills trained within CIPs across schools, just common ones. Course and program content varies across locations to meet local needs limiting the reliability of linkages

• SMEs used were from academic community rather than industry so results may be biased to educators point of view

• Accuracy of deliverables dependent on currency/validity of source crosswalks (e.g., SOC-CIP, DWA-SOC, WECM-CIP crosswalks)

Limitations of Deliverables

27

For any additional information or questions, please contact:

James Loiselle

Research Specialist

Texas Workforce Commission

(512) 491-4974

james.loiselle@cdr.state.tx.us

Contact