Minding the Metacognitive Gap - BSides NOLA

COPYRIGHT © 2016, FIREEYE, INC. ALL RIGHTS

RESERVED.

Chris Sanders (@chrissanders88)

BBQ Pit Master

FireEye/Mandiant

Former DoD &

InGuardian

Founder, Rural Tech Fund

Author

PhD Researcher

Copyright © 2016 Chris Sanders

Disclaimer


I’m going to talk about matters of the brain, not just the normal tech stuff.

My research for this presentation involved consultation with psychologists.

I, however, am not one,

….yet.

Learning Objectives

Increase awareness of:

Metacognitive gap

Investigation process

So you can:

Become a better analyst

Approach investigations in a more systematic way

Get better at training new analysts

Accelerate the effects of experience

Appreciate the value of teaching and learning


The Metacognitive Gap


Perception vs. Reality


Perception

A way of regarding, understanding, or interpreting

something.

Reality

The state of things as they actually exist.

Perception RealityLearning

How do we do it?


How did you learn to catch bad guys?

Experimentation

Observation / OJT

Mentorship

KSU SOC Anthropological Study:

“SOC analysts often perform sophisticated

investigations where the process required to

connect the dots is unclear even to themselves.”

Metacognition

Thinking about thinking

“Why did I do this?”

Understanding your own thought process

Relationship between metacognitive

awareness and performance.

Two Components:

Knowledge of Cognition (Understand It)

Regulation of Cognition (Apply It)


Mapping the Investigation

Process


Experiment Design

Research Questions:

Are experts more metacognitively aware?

What separates novice and expert analysts?

Sample:

Novice and expert analysts

Methodology:

30 case studies

Stimulated recall interviews

Focus on individual investigations of varying types

Perform key phrase analysis


Key Phrase Mapping


Intuition

Experimentation

Restructuring

Imagination

Incubation

Metacognition

Evaluation

Goal Setting

Making Plans

Reflection

Analytically Viewing

Data

Rule-Based

Reasoning

Considering

Alternatives

Dual Process Theory

Intuition: Implicit, unconscious, fast

Reflection: Explicit, controlled, slow

Results


Novices Experts

Intuition Metacognition Reflection

Findings


1. Experienced analysts rely on rule-based

reasoning to a much larger extent.

2. Experienced analysts are more

metacognitively aware than novice analysts.

Closing the Gap


Novice: “How do I do

this job?”

Expert: “Here, watch

me.”

Expert: “Study this way

of thinking. Then,

come try it for

yourself.”

Goal Setting

Making Plans

Evaluation

How can we train analysts to be more

metacognitively aware, and provide them with the

tools to apply that knowledge?

Rule-Based Reasoning


Rule-Based Reasoning


Humans think in if-

then-else

statements

Rules are heuristics

Shortcuts for solving

problems

Derived from

experience

Investigation Heuristics

If the process name is made to look like a

legitimate system process but isn’t

Then it’s probably malware

If the domain has a bunch of random

characters

Then it might have been created by a DGA

Else it’s just a coincedence

If the host is beaconing externally

Then it might be command and control

Else it’s a normal service I should remember for next

time Copyright © 2016 Chris Sanders

Documenting Heuristics


We need an industry wide effort to document these…

If - Then - Else Format

Store in narrative and structured format

Use estimative language

Bonus: You can use these in IR playbooks

Metacognition and the

Investigation Process


The Investigation Process


“An investigation is the systematic inquiry and

examination of evidence and observations in an

effort to gain an accurate perception of whether an

incident has occurred, and to what extent.”

Question

Hypothesis

Answer

Observation

Conclusion

Goal-Driven Questioning

You should be able to articulate what question you’re trying to answer at any given time.

Focus questioning around uncovering relationships

Questioning is driven by rule-based reasoning

Experience really shines here due to a larger library of heuristics

Question

HypothesisAnswer


Hypothesis Generation

You already do this, but

it’s a passive process.

Expose and Attack Bias

Form an educated guess

about the answer to your

questions

Consider your “Because”

statement

I believe X because Y


Question

HypothesisAnswer

Seeking Answers

Key processes:

Finding and Filtering Data

Performing open source intel research

Reviewing evidence

Uncovering additional questions

Hypothesis validation/invalidation


Question

HypothesisAnswer

Investigation Scenario 1

Question

• Was this done maliciously?


Discovery

• SIEM AlertUser account added to domain admin group

Hypothesis

• No – Normal admin activity

Answer

• Yes

Question

• What did the user account do afterwards?

Hypothesis

• Normal admin activities

Answer

• Accessed mail server and mounted exec staff mailboxes

Investigation Scenario 2

Question

• Did the host get infected?


Discovery

• IDS Alert

Angler EK Landing Page

Hypothesis

• Yes

Answer

• No – exploitation failed

Question

• What type of payload was downloaded?

Hypothesis

• Flash exploit due to SWF file alert evidence

Answer

• Hypothesis Confirmed

Question

• Is a vulnerable version of flash installed?

Hypothesis

• It’s Flash, so probably

Answer

• No – Flash is not installed

Further Research


More case studies

Supporting

whitepaper +

dissertation

Further

experimentation in

identified areas

Practical applications

Teaching case

studies

Action Items


Identify and document your rules/heuristics

Start framing through the investigative process

Use the process as a teaching tool

Think about thinking – applied thought has

power

Try to teach this stuff to someone

Thank You!

Web:

http://www.chrissanders.org

E-Mail:

[email protected]

Twitter:

@chrissanders88


Date post:	29-Jan-2018
Category:	Technology
Upload:	chrissanders88
View:	513 times
Download:	0 times

Minding the Metacognitive Gap - BSides NOLA

Technology