1

Lessons Learned using Responsive Virtual Humans for Assessing Interaction Skills

R. Hubal & D. Fishbein, RTI International

M.J. Paschall, Pacific Inst. for Research and Evaluation

2

Overview of Talk

Problem

Experimental Setting

Solution

Methods

Results

Implications

This study was performed under awards # 5-R01-DA14813-02 from the National Institute on Drug Abuse and # EIA-0121211 from the National Science Foundation.

3

Problem

Many inner-city adolescents are at risk during social situations that require:

Emotional control. Conflict resolution through negotiation, seeking

information, expressing preferences. Adverse consequences include violence, school

suspension, criminal activity. Training on these skills is well studied, but…

Assessment is unsatisfactory:

Want to assess adolescent’s learning before he’s “on the street”.

Existing methods use surveys or text-based cases, not “assessment-by-doing”.

Problem Experimental

Setting Solution Methods Results Implications

4

Experimental Setting

Part of a larger study:

Examining neurocognitive-emotive predictors of behavioral problems among minority adolescents in high-risk urban settings.

Identifying promising approaches to improve social-cognitive skills and reduce problem behaviors such as interpersonal violence and drug abuse.

Participants part of an ongoing longitudinal study.

Research team includes RTI, JHU, PIRE, others.

Problem Experimental

Setting Solution Methods Results Implications

5

Solution

Role playing is known to be effective in providing adolescents with social competency skills.

Use responsive virtual characters to simulate role-play scenarios for assessment:

Interactive 3D.

Natural language processing.

Cognitive, social, emotional models.

Problem Experimental

Setting Solution Methods Results Implications

6

Solution, continued

Responsive virtual human training applications:

Law officers learning to manage encounters with the mentally ill.

Medical practitioners practicing on virtual adult and pediatric standardized patients.

Survey field interviewers learning to avert non-response and to gain informed consent to participate.

Army ground tracked vehicle mechanics using a maintenance training system.

These were not intended for assessment…

Problem Experimental

Setting Solution Methods Results Implications

7

Solution, continued

Vignette storylines designed to target skills (engaging in negotiation, maintaining emotional control and being non-provocative, seeking information).

Language models designed to interpret participant input as reflecting one or more social skill.

Behavior models designed so that the virtual character attempts different strategies to require participants to demonstrate skills.

Methods devised to measure participant engagement.

Problem Experimental

Setting Solution Methods Results Implications

8

Methods

Participants:

Black male inner-city 10th graders.

Approx. half diagnosed with Conduct Disorder.

Materials:

PACT video “Workin’ It Out”.

Virtual character vignettes:– 3 scenarios (stolen goods, invitation,

confrontation).– 2 variations (pre/post).

Behavioral reports, teacher ratings, self-reports.

Problem Experimental

Setting Solution Methods Results Implications

9

Problem Experimental

Setting Solution Methods Results Implications

Methods, continued

Positive Adolescents Choices Training (PACT) content:

Social skills training program developed specifically for black youth.

Uses videotapes containing vignettes of a conflict situation and demonstrating skills that could defuse a potentially violent situation.

Target skills include:– Giving negative feedback or expressing criticism calmly

(“Givin’ It”).– Reacting appropriately to the criticism or anger of others

(“Takin’ It”).– Identifying problems and solutions (“Workin’ It Out”).

10

Methods, continued

Procedure:

Participant given pre-test set of vignettes.– Wizard-of-Oz methodology.– Monitor how participants handle vignettes designed to

elicit risky behavior.

Participant given standard text-based assessments.

Participant shown PACT video in presence of facilitator.

After up to 6 month delay, participant given post-test set of vignettes (again WoZ).

Behavioral reports, teacher ratings, self-reports collected.

Problem Experimental

Setting Solution Methods Results Implications

11

Methods, continued

Measures:

Engagement with vignettes.

Body language.

Verbalizations.

Number of conversational turns.

Response time.

Outcome:– Positive outcome is to decline or back away.– Negative outcome is to agree or escalate

confrontation.

Problem Experimental

Setting Solution Methods Results Implications

12

Results

Scoring procedures used during pre/post test with vignettes identify:

Level of emotional control.

Interpersonal communication skills.

Analysis against established measures provides some support for construct and criterion validity.

Nearly all queried participants stated their virtual decision mirrored what would be their real-life decision.

Note: acceptability / usability not different among groups.

Problem Experimental

Setting Solution Methods Results Implications

13

Vignette Outcome

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

PACT noPACT PACT noPACT

CD control

session 1

session 2

Vignette Outcome

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

PACT noPACT PACT noPACT

CD control

session 1

session 2

Participants agree to help/go along with/escalate confrontation with the synthetic character one-third to half of time.

Simulation effective in differentiating:

Adolescents with clinical Conduct Disorder.

Adolescents who had participated in live training sessions on key skills.

Results, continued

Problem Experimental

Setting Solution Methods Results Implications

14

Implications

Virtual characters can be useful for assessment as well as training in some contexts.

Military applications:

Greater understanding of signs and symptoms associated with PTSD.

Managing encounters with the mentally ill.

Interaction with child casualties and combatants.

Crowd control or checkpoint management.

Essentially, where need is to assess interaction skills in environment that is costly, dangerous, or difficult to replicate.

Problem Experimental

Setting Solution Methods Results Implications

15

Questions?

This study was performed under awards # 5-R01-DA14813-02 from the National Institute on Drug Abuse and # EIA-0121211 from the National Science Foundation.