Pal Thomassen
Software securityState of the theory vs state of thepractice
Fall project, 2011
Faculty of Information Technology, Mathematics and Electrical EngineeringDepartment of Computer and Information Science
Contents
1 Introduction 21.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Report Structure . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Preliminary Studies 42.1 What is Software Security . . . . . . . . . . . . . . . . . . . . 42.2 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Finding the State of the Art . . . . . . . . . . . . . . . 52.3 The Four Biggest Frameworks . . . . . . . . . . . . . . . . . . 7
2.3.1 Microsoft Security Development Process . . . . . . . . 72.3.2 CLASP - Comprehensive, Lightweight Application Se-
curity Process . . . . . . . . . . . . . . . . . . . . . . . 112.3.3 Seven Touchpoints . . . . . . . . . . . . . . . . . . . . 172.3.4 BSIMM . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Comparison of the Frameworks . . . . . . . . . . . . . . . . . 262.4.1 Survey Activities . . . . . . . . . . . . . . . . . . . . . 28
2.5 Summary of Preliminary Studies . . . . . . . . . . . . . . . . 292.5.1 Research Question 1 . . . . . . . . . . . . . . . . . . . 29
3 Research Process 303.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . 303.2 Research Process . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.1 Quantitative Research Strategy . . . . . . . . . . . . . 313.2.2 Qualitative Research Strategy . . . . . . . . . . . . . . 313.2.3 Triangulation . . . . . . . . . . . . . . . . . . . . . . . 313.2.4 Research Strategy . . . . . . . . . . . . . . . . . . . . 323.2.5 Cross-sectional methods . . . . . . . . . . . . . . . . . 33
3.3 Survey Construction . . . . . . . . . . . . . . . . . . . . . . . 343.3.1 A note about survey lengths . . . . . . . . . . . . . . . 353.3.2 Initial Survey Construction . . . . . . . . . . . . . . . 363.3.3 Survey trimming . . . . . . . . . . . . . . . . . . . . . 37
3.4 Projects to Study . . . . . . . . . . . . . . . . . . . . . . . . . 383.4.1 Consultant companies . . . . . . . . . . . . . . . . . . 38
I
3.4.2 Non-consultant companies . . . . . . . . . . . . . . . . 39
3.5 Summary of research process . . . . . . . . . . . . . . . . . . 39
4 Activities in the Survey 40
4.1 Practical Information . . . . . . . . . . . . . . . . . . . . . . . 40
4.2 Training/Education . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3.1 Abuse Cases . . . . . . . . . . . . . . . . . . . . . . . 42
4.3.2 Security Requirements . . . . . . . . . . . . . . . . . . 42
4.4 Design and Planning . . . . . . . . . . . . . . . . . . . . . . . 43
4.4.1 Architectural Risk Analysis . . . . . . . . . . . . . . . 43
4.4.2 Attack Surface . . . . . . . . . . . . . . . . . . . . . . 44
4.4.3 Security Patterns . . . . . . . . . . . . . . . . . . . . . 44
4.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.6 Testing and Verification . . . . . . . . . . . . . . . . . . . . . 46
4.6.1 Penetration testing . . . . . . . . . . . . . . . . . . . . 46
4.6.2 Fuzz-testing . . . . . . . . . . . . . . . . . . . . . . . . 46
4.7 Release and Maintenance . . . . . . . . . . . . . . . . . . . . 47
4.7.1 Security Operations . . . . . . . . . . . . . . . . . . . 47
4.8 Summary of Survey Activities . . . . . . . . . . . . . . . . . . 47
5 Results 48
5.1 Survey results . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.2 Discussion of the survey results . . . . . . . . . . . . . . . . . 48
5.2.1 Project sizes and time spans . . . . . . . . . . . . . . 49
5.2.2 Other practical information about the participants . . 49
5.2.3 Evaluation of COTS . . . . . . . . . . . . . . . . . . . 49
5.2.4 Results of many yes and no answers . . . . . . . . . . 50
5.2.5 Threat modeling . . . . . . . . . . . . . . . . . . . . . 50
5.2.6 The activities that almost none did . . . . . . . . . . . 51
5.3 Results validity . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.4 Interview results . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.5 Summary of the interview results . . . . . . . . . . . . . . . . 52
5.5.1 Security review process at one company . . . . . . . . 53
6 Analysis 54
6.1 General trends in the survey . . . . . . . . . . . . . . . . . . . 54
6.2 Correlation between different activities . . . . . . . . . . . . . 56
6.2.1 Pearson product-moment correlation coefficient . . . . 56
6.2.2 The usage of Pearsons r . . . . . . . . . . . . . . . . . 57
6.2.3 Correlations . . . . . . . . . . . . . . . . . . . . . . . . 59
6.2.4 Fuzz testing, odd results . . . . . . . . . . . . . . . . . 59
6.2.5 Penetration testing, an entry gate . . . . . . . . . . . 59
6.2.6 Reactive security, the most dominant . . . . . . . . . . 60
II
6.2.7 Binary approach, either you do it, or you don’t . . . . 606.2.8 Answering RQ2 . . . . . . . . . . . . . . . . . . . . . . 61
6.3 Which phases is the focus on . . . . . . . . . . . . . . . . . . 626.3.1 Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . 626.3.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . 626.3.3 Requirements and design and planning . . . . . . . . . 626.3.4 Implementaion . . . . . . . . . . . . . . . . . . . . . . 636.3.5 Verification and testing . . . . . . . . . . . . . . . . . 636.3.6 Summary of focus during the phases . . . . . . . . . . 64
7 Conclusion 657.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . 65
7.1.1 Research question 1 . . . . . . . . . . . . . . . . . . . 657.1.2 Research Question 2 . . . . . . . . . . . . . . . . . . . 66
7.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.3 Further work . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Appendix A Survey Results i
Appendix B Correlations table viii
Appendix C Interview Results xiC.1 Project information . . . . . . . . . . . . . . . . . . . . . . . . xiC.2 Training and education . . . . . . . . . . . . . . . . . . . . . . xiiC.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiC.4 Design and planning . . . . . . . . . . . . . . . . . . . . . . . xiiiC.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . xivC.6 Testing and verification . . . . . . . . . . . . . . . . . . . . . xviC.7 Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Appendix D Survey xviii
III
List of Tables
2.1 List of activities in CLASP . . . . . . . . . . . . . . . . . . . 152.2 The Software Security Framework(SFF) . . . . . . . . . . . . 23
3.1 Research strategies, from Ringdal [Rin09] . . . . . . . . . . . 323.2 Cross sectional methods . . . . . . . . . . . . . . . . . . . . . 33
6.1 Strength of Pearsons r . . . . . . . . . . . . . . . . . . . . . . 586.2 Strong correlations . . . . . . . . . . . . . . . . . . . . . . . . 61
A.1 Results from the survey . . . . . . . . . . . . . . . . . . . . . i
B.1 Correlations of Yes and No answers . . . . . . . . . . . . . . . ixB.2 Correlations of Yes and No answers continued . . . . . . . . . x
IV
List of Figures
2.1 The SDL simplified . . . . . . . . . . . . . . . . . . . . . . . . 72.2 CLASP views . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3 Seven touchpoints . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1 How the survey was originally made on a whiteboard. Thearrow show the order of the question groups . . . . . . . . . . 37
5.1 Project size and time span . . . . . . . . . . . . . . . . . . . . 495.2 Penetration Testing, Risk Assessment, Attack Surface analysis
and Security Requirements results . . . . . . . . . . . . . . . 50
6.1 Answers to: Do you have specific security requirements? . . . 556.2 Answers to: Do you perform a risk assessment to determine
the biggest risks for the software? . . . . . . . . . . . . . . . . 556.3 Correlation of different value of r . . . . . . . . . . . . . . . . 576.4 Confidence interval showing both tails . . . . . . . . . . . . . 576.5 How many types of training each respondent had . . . . . . . 63
V
VI
Abstract
This specialization project presents how the industry in Norway practicessoftware security and compare this with state of the art research and theory.
The project made a web-based survey and got several companies from theindustry to participate. This was used as a quantitative data source andto support the findings from the survey interview was done with severalof the participants for verifying qualitative data. The report explains howthe survey was constructed, and the results from both the survey and theinterviews.
The most interesting discovery was that for a software project its use ofsoftware security activities was really binary. Either the project did a greatdeal of activities or the project did almost none. However all the projectshad a way of fixing vulnerabilities after released which is the more classicalreactive model.
The project took time from August to December 2011.
Acknowledgements
I would like to thank all the people who reviewed the survey for errors, bothsyntactical and logical errors. These people include, Magnus Sellereite Fjell,Stian Veum Møllersen, Martin Sandsmark, Ken Børge Viktil, Trond Klakkenand Asmund Eldhuset.
Tor Stalhane for tips on how to create a survey and lending a book abouthow to do statistical and scientific studies.
My supervisor Lillian Røstad for help with both constructing the survey, thereport and general help, tips and motivation during the whole project.
Lastly I want to thank my wife Stine Mari Enlien Thomassen, for beingsupporting and understandable of the time required to finish the project.
Chapter 1
Introduction
1.1 Motivation
Software Security have been under research for some years, but the industryis still catching up with the idea of building software more robust and secure.There have been many major players in the field being more focused onbuilding more secure and robust software for example the department ofhomeland security in the United States1 or companies as Microsoft with theirtrust worthy computing initiative2, however the idea is still quite new. Theproject will therefore see how the Norwegian software industry have caughtup with the idea of software security.
The overall goal will therefore be to find out how the industry in Norwaythreats software security during the development lifecycle of software andcompare it with state of the art theory and research.
Too see exactly how much the industry in Norway is thinking about softwaresecurity when they are developing software, the project have conducted asurvey covering a couple of companies in Norway. The study have comparedthe current state of software security with state of the art methodologies andframeworks. To help with the overall goal, the following research questionswhere made:
• RQ1: What are the theories and state of the art research available forsoftware security?
• RQ2: How is software security done in practice in Norwegian industrycompared to the state of art research and theory
1https://buildsecurityin.us-cert.gov/bsi/home.html2http://www.microsoft.com/about/twc/en/us/default.aspx
2
– RQ2.2: Does software security have the same focus in every stageof an applications lifecycle
These questions will help in finding the overall goal stated above, andnaturally divides the task in two, one to identify what the current state ofthe art of software security is, and the other part which is to compare thattheory with the practice in the industry.
Some of the companies which participated in the survey are internationalbut this study will only include the Norwegian division or offices, since thatis the part of interest. To observe how the current state of software securityis in the development process of different companies we have done botha survey and interviews. To narrow down the study, only projects whichdevelop software that are exposed to the public were included. The reason isthat software hidden behind a firewall often have less requirement for goodsecurity, but relies on firewalls, intrusion detection/prevention systems andso on to keep it secure.
1.2 Report Structure
The report consists of seven chapters. Chapter 2 go through the findings froma small literature study performed. Chapter 3 go through the research processused to answer RQ2 and construct the survey and chapter 4 go thorugh thequestions selected for the survey. Chapter 3 also highlight why a web basedsurvey was chosen compared to other similar research methods. The nextchapter present the results from the survey and interviews. Only interestingfindings are highlighted here and the complete results is in appendix A.Chapter 6 discusses how the survey results were analyzed and how thestatistical analysis were performed. The final chapter is the conclusion andcontains what further research and work is required.
3
Chapter 2
Preliminary Studies
This chapter describes the current state of the art research and theory forsoftware security. To begin it will present a definition of software securitywhich is used in this report. Second it will go trough the state of the art inthe field of software security. The report contains a lot of work done by theindustry and different software companies in addition to academic papers.This work is also what is used to answer RQ1.
2.1 What is Software Security
There are many definitions of software security. In department of homelandsecurity discuss in [IAT07] many definitions which have been used by variousresearchers and industry leaders. Many definitions comes from softwareassurance, which is more about reliability and quality with security beingbaked into those. Microsoft have a strategy called trustworthy computingwhich also incorporates software security. All these definitions are quiteadvanced and this report will therefore us McGraw’s simple definition fromhis book [McG06]. It is presented below.
Software security is the idea of engineering software so that itcontinues to function correctly under malicious attack
The definition does not have anything to do about perimeter security which isdifferent security mechanisms around a software system. It is about buildingsoftware that can withstand most attacks and function normally despiteefforts to exploit the software without any perimeter security mechanismsin place. This report will because of this not focus on perimeter securitymechanisms but rather on how to build the software itself secure. However aswe will point out that perimeter security is important, but modern software
4
often have a web-interface meaning that there is some parts of the softwarewhich must be exposed to attackers regardless of perimeter security.
2.2 State of the Art
This section covers the state of the art research and theories in the softwaresecurity field. It is not a comprehensive literature study, but is sufficient toanswer RQ1. Our goal was to identify the biggest and most used frameworksfor doing software security in the development lifecycle and use the mostcommon activities from those to construct a survey. The section starts withhow the most used frameworks where identified and found, then presentseach framework and it’s characteristics.
What was searched for was frameworks, guidelines or activities that eithercould be used with an existing software development process, or act as it ownsoftware development process but with a big focus on security. The searchrevealed many smaller papers which discussed a few activities more isolatedand some big frameworks which contained many activities and covered thewhole development lifecycle of a software application. The main focus wason the big frameworks since they are most used in the industry.
2.2.1 Finding the State of the Art
In order to find the state of the art frameworks, several resources whereused. One large report from the Department of Homeland Security[IAT07]listed several Security Lifecycle frameworks, the downside was that it waswritten in 2007, so the information was a bit dated. This report was used asa starting point, and all the frameworks mentioned where researched. Therewhere some some frameworks which where mentioned in the [IAT07] reportbut when looked into they dit not contain anything useful, these are listedbelow.
• TSP-Secure
• Oracle Software Security Assurance Process
When searching for TSP-secure the only results was a presentation from2002 on the subject, and all other searches for TSP-secure did not goanywhere, hence it was not included in the state of the art research. Thesecond framework which were researched was the Oracle Software SecurityAssurance process, however all that where found was one whitepaper[OP07]which contained little useful information. The other three that where mentionin the department of homeland security report was all available, and hence
5
these three where included. These three was Microsofts Security DevelopmentLifecycle, McGraw’s seven touchpoints and OWASP CLASP.
The search for information about the three frameworks found a more recentpaper comparing these three [DSB+09]. This paper further acknowledgedthat these three where some of the bigger and most used frameworks fordeveloping secure software. An IEEE Paper[CA11] regarding the practiceof software security mentions the BSIMM framework as a new framework.Further research into this frameworks reveals a website1 with a thoroughdescription of the BSIMM framework in the paper[MCM09]. The BSIMMframework is quite new, so we did not find any other papers written aboutit other than it own paper from 2010 [MCM09]. BSIMM consists of manyauthors especially McGraw which also has the seven touchpoints and thebook [McG06] about software security. Another interesting point aboutBSIMM is that in addition to being a more recent framework than the otherthree, it also has a different take on some of the topics which we will explainlater. This earned it a place among the frameworks which were considered tobe one of the big, complete software security frameworks despite few otherreferences to it, the main reason for this is probably because it is quite newcompared to the others.
When searching for information about Microsoft SDL, OWASP CLASP andMcGraws Touchpoints several other papers mentioning them, and two whichcompared them. One paper [SSSB08] also compared these three and theCorrectness by Construction framework [HC02]. The framework called CbyCin the paper was a framework which did not find much information aboutsave for one paper by Hall and Chapman [HC02]. Because of this it wasnot included in the frameworks considered to be complete and widely used.The other three, SDL, CLASP and Touchpoints were all widely cited andseemed to be much used in the industry both by reading the frameworksown claims and also by them being widely cited and used in different papers,for example [HC02] and [SSSB08].
Further research into the topic did not reveal any more frameworks and itwas concluded that these frameworks where the biggest and most used onesin the field of software security. They where the basis for the construction ofthe survey.
• Microsoft Security Development Process
• Comprehensive, Lightweight Application Security Process (CLASP)
• Seven Touchpoints by Gary McGraw from Cigital
• Building Security In Maturity Model (BSIMM)
1http://bsimm.com/
6
The frameworks will be presented below and compared to eachother. Thiswill also reflect how the different frameworks are focusing on security duringdifferent application life cycles.
2.3 The Four Biggest Frameworks
This section will walk through the four big frameworks which were identifiedabove. The focus will be on the general characteristics of the frameworksand also highlight interesting point about each framework. Chapter 4 willwalk through how the survey questions was selected with the basis in theseframeworks. This section will serve as a basis for answering research questionRQ1.
2.3.1 Microsoft Security Development Process
Microsoft’s Security Development Process from here on called the SDL isa methodology used and developed by Microsoft and many of it’s partners.The project contains one book[HL06] and several other resources. In thisreport the simplified implementation[Mic10] and the process guide [Mic11]was used as the main sources. The SDL consists of seven main areas whichare different stages in a traditional waterfall development process. The SDLcan also be used with an agile development methodology and is called SDLAgile. The SDL is unique in that it is a complete development lifecycle andspecifies exactly how it is to be implemented.
The different SDL activities are summarized in figure 2.1. These differentsecurity activities are tied to different phases of the development process.The SDL also have a section about SDL in an Agile methodology. Theactivities listed in the figure are all required activities to be done using theSDL, there are also additional optional requirements which can be performedat various stages in the SDL. The SDL is a large framework, so figure 2.1 isa simplified view of the SDL.
Figure 2.1: The SDL simplified
7
Characteristics of the SDL
The SDL has as its primary goal to increase the quality of functionality-drivensoftware by improving its security posture. The activities are often tied toactivities which generate new functionality. For example, all features shouldbe reviewed to determine if it should be enabled by default or not, this is toreduce the attack surface as much as possible while developing new features.
The SDL is really specific with the usage of tools and how to perform thedifferent activities. For example it contains specific tools which are to beused for static code analysis, it contains it’s own list of banned API’s whichshould not be used in a programming project. Many of the other frameworksmentions that tools should be used but places no restriction on which toolsto be used. This makes it easier to start using the SDL since it provide goodguidance for how to perform the different activities.
Phases of the SDL
This subsection walks through the phases of the SDL and highlights activitieson the way.
The SDL starts by listing training as one of it’s core activities. The SDLestablishes a baseline which every member of a software development teammust have as well as advanced training which only some people involved inthe project need. It then goes on to describe the different activities usedduring the different stages in software development, these activities are listedin figure 2.1.
In the requirements phase it lists security requirements which is commonamong all the frameworks, in addition when reading [Mic11] we can seethat the SDL also has a big focus in privacy. This is unique among theframeworks since many of the others focus more on security and not somuch on privacy, while the SDL treats both as important. Another uniquething in this phase is the establishment of quality gates/bug bars. Theseare quality requirements which need to be set early in the project and neverrelaxed. One example of a quality gate from the SDL is, a critical bug whichis something that is really import that it gets fixed as soon as possible, thesoftware cannot be released before this bug is fixed. A critical bug can besomething that allow an attacker to elevate privileges and execute arbitrarycode or obtain more privileges than intended. At last the SDL contains arisk assessment, the interesting point here is that it contains both a securityand privacy risk assessment.
In the design phase the SDL mentions design requirements. Here it mostlymeans design requirements that have a focus on security. An example of
8
a security requirement from the SDL process guide [Mic11] is to use theuser Account Control system for Windows Vista and newer. Next we haveboth analysis of attack surface and threat modeling, these activities will bedescribed in greater detail further down in section 2.4.
The implementation phase of the SDL have a lot of information about tools.Both which tools is approved to be used with the SDL as well as certaincompiler options that makes the software more secure. One point is codereview, both manually and with a static analysis tool. In the figure 2.1 onlystatic analysis in mentioned, but in the process guide [Mic11] code reviewin general is mentioned. Code review can both be done manually, meaningdevelopers reading the code, checking it against a checklist for things to lookafter as well as automated with the usage of a static analysis tool.
The next point in figure 2.1 is verification, this is also unique to the SDLsince most other frameworks call this phase testing. In this phase the SDLhave many review activities in addition to the traditional testing which allthe other frameworks contains. The review phase also contains code review,but this time it is mostly mentioned as manual code review. The reviewstage also involves reviewing old threat models as well as review that thesethreat models still apply and see if they changed during the implementation.The SDL also contains an activity they call the security push. The securitypush involves the whole development team with updating threat models,code review, testing and thorough documentation review and edit. Thesecurity push is something that should be done once most of the coding andfunctionality is complete. A security push is not an excuse to not do anyreview before this stage, but as the SDL points out this can happen aroundthe same time as the software enters beta testing and the focus is on qualityassurance.
The last parts of the SDL is to make an incident response plan. This is aplan which tells the team how to handle mistakes and security vulnerabilitiesdiscovered after the software have been either released to web or released tomanufacturing. The plan should contain guidelines both for how to patcha vulnerability but also how to classify the vulnerability found in terms ofseriousness. In addition there is a final security review which is some shortof acceptance test which the application have to pass for it to be released.Again this activity includes review of both code, diagrams and threat modelsas well as testing.
As we can see the SDL includes activities for every stage of the softwaredevelopment lifecycle. This is a common theme among many of theseframeworks and drills down the point of having a focus on security throughthe entire lifecycle, not only during the testing phases. One interesting thingabout the SDL is the inclusion of Fuzz testing. All the other literature andframeworks almost don’t mention Fuzz testing but Microsoft has listed is as
9
one of their required tasks. Most other frameworks instead lists penetrationtesting as a required activity, but Microsoft lists penetration testing asoptional.
SDL-Agile
The SDL-Agile is an agile version of the SDL. Since much software developedtoday is using an Agile methodology Microsoft has addressed the issue thatthe traditional SDL is to heavyweight to use in an Agile methodology. AsMicrosoft sees the sprint as the workhorse of agile methods. The SDL Agileis therefore focused around sprints. SDL Agile have a set of requirementsfor each sprint with different categories of requirements. Every sprint re-quirements which are to be performed every sprint, bucket requirementswhich do not need to be done every sprint and at last one-time requirementswhich are done once. The every sprint requirements are said to be the mostimportant activities for developing secure software, and therefore should beincorporated into every sprint. An example of these requirements are: Runanalysis tools daily or per build, threat model all new features, ensure thateach project member has completed at least one security training course,there are many other activities which are mentioned in [Mic11].
The next category, the bucket requirements are not important enough to bemandated for every sprint, but should still be done quite often. The bucketcategory is subdivided into three separated buckets of related tasks. Currentlythere are three buckets in the buckets category, verifications tasks (mostlyfuzzers and other analysis tools), design review tasks and planning tasks.The team performing the SDL-Agile must complete one of the requirementsfrom the related buckets each sprint. The team is left up to their own todetermine which tasks are to be performed when.
The last category is the one-time requirements. This are as the name impliesonly done one-time per project. Since some of these tasks can span overmore than one sprint, they are allowed a grace period which means that therequirement can span across multiple sprints.
The SDL-Agile have some constraints with regards to both project time andsprint durations. A project with a two or three year completion time canbe considered to perform all the requirements of the SDL-Agile, howeverprojects which span over the course of a few months will not have time tocomplete all the steps. The SDL team believes that this provides a good mixof security, feature development and speed of release.
10
Summary Microsoft Security Development Lifecycle
The SDL process at Microsoft contains many common activities shared withthe other frameworks for software security. One of the biggest differencesis the focus on Fuzz testing which other frameworks does not have, or atleast as a minor activity. In the SDL Penetration testing is an optionalactivity that can be used to verify that the security of the software havebeen addressed properly. In for example the seven touchpoints from McGraw[McG06] a penetration test is used instead of a final security review to verifythat the security have been addressed properly in the project.
Another big difference is that the SDL contains much more specific infor-mation about tools and how to perform the activities. Many of the otherframeworks makes it up to the user of the framework to tailor the details totheir own project, while the SDL more clearly defines how activities shouldbe done. This is both an advantage and disadvantage. It is an advantagesince it makes it easier to adopt the SDL because of its good guidance inhow and what to use. On the other side it is a disadvantage since its notsuitable for projects which do not use Microsoft technology.
Microsoft’s SDL also takes into account different development methodolo-gies, since it is a development lifecycle in itself. While both CLASP andthe touchpoints are methodology agnostic, none of them describes how toincorporate their activities into an Agile development methodology while theSDL is already suitable for both waterfall and Agile projects.
2.3.2 CLASP - Comprehensive, Lightweight Application Se-curity Process
CLASP is a development methodology agnostic software security framework.It contains 24 activities which can be used in different places in the devel-opment process, regardless of methodology. CLASP also supports a slowadoption of the framework so that a development team can start with a fewactivities and add more later. To support this CLASP contains a descriptionof the dangers of omitting a certain activity so that the team will be awareof the risks they are running. In addition CLASP also relies heavily uponrisk assessment to be done so that the activities can be chosen based on thespecific projects risks. CLASP is now an OWASP project but was originallydeveloped by Secure Software. The most recent version of CLASP is version1.2 from the OWASP CLASP website[Lig06]. CLASP focuses on leveraginga database of knowledge, such as the CLASP vulnerability lexicon with itsactivities.
In this report the focus is mostly on the role and activitiy based views of
11
CLASP. Since the vulnerability lexicon is really large and extensive andmeant to be used as a support when performing some of the CLASP activities,it was therefore chosen to not focus much on it but mention the role it playsin CLASP.
Overview of CLASP
CLASP consists of several views detailed below, however in a birds eye viewCLASP consist of a set of roles which defines stakeholdes for a set of securityactivities. These activities in turn relies on the vulnerability lexicon definedin CLASP. CLASP can be seen by five different views, which are shown inthe list below. These views are different ways of looking at CLASP anddescribes the different components which makes up the CLASP process.
• Concepts view
• Role-Based view
• Acitivitiy Assessment view
• Activity Implementation view
• Vulnerability view
Figure 2.2 shows the different views and how they relate to each other inCLASP. We will not go into CLASP in detail here but give a brief overviewof the different roles and activities of CLASP.
Role Based View
The role based view of CLASP is all the different roles that CLASP contains.CLASP contains a set of roles that a project should have. In the book aboutCLASP there is a table over all the roles and their corresponding activities.Some roles are bigger than others and cover many activities and some onlycover one or a few activities. The reason behind these different roles is toshow who is responsible for what, but also to make it easy to insert CLASPinto an existing team where many of these roles or similar roles already exist.The biggest role in CLASP is the project manager, which is responsible fordriving the CLASP initiative in the project. In table 2.1 all the activitiesare listed with their related roles in the project.
One of the challenges with CLASP is that a team has to assignt these roles,for a big project which already have many of these roles this would not be aproblem, however for a smaller project with only a handful of developers itbecomes significantly harder to assign the roles to different team members.This can result in that the roles being mostly artificial and does not reflect
12
Figure 2.2: CLASP views
how the team is actually organised, perhaps the developers themselves areboth the architects, integrator and requirements specifier. This can be seenas both a strength and a weakness of CLASP. It is a strength because it helpsteams which already have similar roles in their exisiting development lifecycleto figure out who is responsible for which activity. It can be a weakness forsmaller teams since they need to make these roles in order to find out who isresponsible for which activity.
Activities Based View
CLASP contains a set of activities that are performed at different stagesduring development of software. The CLASP process is agnostic to method-ologies so these activities can be applied both to traditional waterfall projectsand to more agile projects which typically will do some of these activities
13
often but not in the same scale as those for the waterfall projects. It is upto the implementer of CLASP to decide how this is done.
CLASP have a total of 24 activities, but they say that it is not an all ornothing thing. An organization can just focus on a few activities that givesmost benefit-for-cost and later add more as time goes by. CLASP alsoincludes with each activity an Information on activity applicability, meaningwhere in the development process the activity can be used. Discussion ofrisks associated with omitting the activity describes in short what risks thereare if a certain activity is omitted. Since CLASP activities can be done as fewor as many as one would like it makes sense to describe the risk of omittingan activity. Lastly they also include An indication of implementation costfor each activity, that way it’s easy to calculate which activities will bringthe most benefit-for-cost to a project.
CLASP starts with activities that are performed early in the software devel-opment lifecycle. For example the institute awareness program is to educateand train team members about software security. CLASP also suggests thatthe project should contain a security responsible for the project. Monitormetrics can look a bit like the SDLs bug bars and quality gates, however, thispoint also means monitoring metrics which can for example be the attacksurface of the application so it covers all the metrics in a project.
When it comes to requirements CLASP has several activities. One is tospecify the operational environment, document security requirements, identifyresources and trust boundaries and misuse cases. A interesting point herewhich covers the software developer organization in its whole is the globalsecurity policy activity, which is a security policy determined to be usedglobally in the organization and shared between projects. This activity isquite unique since it covers the organization as a whole, however, since thesurvey produced as part of the specialization project focused on individualprojects and not the organization as a whole, it was difficult to include it inthe survey. The security requirements and misuse cases in this phase is verysimilar to the same activities in the SDL.
CLASP have several activities which would tie into a design phase of awaterfall structured software project. Many of these activities are quitesimilar to the SDLs activities in the same phase. These include, identifyattack surface, apply security principles to design and research and Performsecurity analysis of system requirements and design (threat modeling). Whenwe look closely at the activity in the CLASP activity implementation we seesimilarities between apply security principles and the SDLs threat modelingand research and assess security posture of technology solutions and the SDLsrisk assessment. However the CLASP assess security posture of technologysolutions focus a lot on COTS components, while the SDL mentions COTScomponents CLASP have a big focus in reviewing and assess the risks of
14
Table 2.1: List of activities in CLASPCLASP actitvity Related Project Role
Institute security awareness program Project Manager
Monitor security metrics Project Manager
Specify operational environment Owner: Requirements specifier, Key con-tributor: Architect
Identify global security policy Recuirements specifier
Identify resources and trust boundaries Owner: Architect, Key contributor: Re-quirements specifier
Identify user roles and resource capabilities Owner: Architect, Key contributor: Re-quirements specifier
Document security-relevant requirements Owner: Requirements Specifier, Key con-tributor: Architect
Detail misuse cases Owner: Requirements specifier, Key con-tributor: Stakeholder
Identify attack surface Designer
Apply security principles to design Designer
Research and assess security posture oftechnology solutions
Owner: Designer, Key contributor: Com-ponent Vendor
Annotate class designs with security prop-erties
Designer
Specify database security configuration Database designer
Perform security analysis of system require-ments and design (threat modeling)
Security Auditor
Integrate security analysis into source man-agement process
Integrator
Implement interface contracts Implementer
Implement and eloborate resource policiesand security technologies
Implementer
Address reported security issues Owner: Designer, Fault Reporter
Perform source-level security review Owner: Security Auditor, Key contributor:Implementer, designer
Identify, implement and perform securitytests
Test analyst
Verify security attributes of resources Tester
Perform code signing Intergrator
Build operational security guide Owner: Integrator, key contributor: De-signer, Architect, Implementer
Manage security issue disclosure process Owner: Project manager, Key contributor:Designer
15
using them in the software project. In addition CLASP have some specificactivities such as specify database security configuration, this activity impliesthat there is a database component in the software system which is notalways certain.
In the implementation phase CLASP promotes the use of both dynamicand static code analysis tools in its integrate security analysis into sourcemanagement process. This activity coupled with source-level security reviewhave much in common with the SDLs review phase which consists of reviewof both requirements as well as source code. CLASP also contains anactivity which is to implement and elaborate resource policies and securitytechnologies, this activity is quite vague in its description but it seems toensure that coding guidelines are met.
When it comes to testing CLASP have several activities. The biggest activityhere is the identify, implement and perform security tests. This activityinvolves both testing security functionality and penetration testing. Thisstage does not involve as much review as the SDLs verification phase butinvolves more testing, but also some verification especially of the operatingenvironment. The activity verify security attributes of resources is to gosystematically through the operation environment and test and verify thesecurity of it.
The last activities of CLASP involves the release of the software. Just aswith SDL, CLASP also include code signing and a way of fixing vulnera-bilities which are discovered after deployment. One interesting activity ofCLASP is the build operational security guide. This activity means that thestakeholders get a operational guide of the software which helps them useand deploy the software securely.
CLASP Vulnerabilities
CLASP in common with the book Software Security: Building the securityin[McG06] also contains a taxonomy for vulnerabilities and exploits. Incontrast to [McG06] the CLASP taxonomy is quite comprehensive and large.This taxonomy is meant to help developers when doing a manual code reviewor testing in order to classify the vulnerabilities discovered. In additionthe CLASP Vulnerability Lexicon also helps with the risk associated witha vulnerability, helping in making a risk assessment and also to see theseverity of the vulnerability. CLASP not only contains a taxonomy but alsohave a lexicon with many concrete vulnerabilities. The vulnerability lexiconcontains a list of classifications which make up the high level taxonomy ofthe vulnerabilities. This lexicon is meant to be used extensively throughthe whole implementation of CLASP and being used as a shared resource
16
between many activities.
The vulnerability lexicon in CLASP is quite unique compared to the otherframeworks presented. The SDL contains many specific tools and often helpsto guide the users of the SDL for how to perform the different activitiesspecifically, but it does not contain a lexicon in the same way as CLASP.
Summary of CLASP
The CLASP process tries in contrast to the other processes try to leveragea database of knowledge to be used, the vulnerability lexicon. In commonwith the other processes CLASP have a set of roles and activities whichare to be performed during a software development lifecycle, but CLASPhave a bigger focus on roles. CLASP do not specify where in the softwaredevelopment cycle the activities should be used, nor mentions anything aboutdevelopment methodologies. This makes it up to the project manager andteam to decide how to integrate the CLASP activities and roles into theirexisting methodology either it be Agile, waterfall or lean.
CLASP also have different names for many of the activities which still arevery familiar with the other frameworks, this makes it a bit harder to compareCLASP with the others and also identify when the activities are practicallythe same and when there are significant differences. In the rundown it wascompared extensively with the SDLs similar activities.
2.3.3 Seven Touchpoints
McGraw’s seven security touchpoints are seven different activities whichtouches different artifacts in the development process, hence making themmethodology agnostic just like CLASP. McGraw also have an order ofeffectiveness of the touchpoints, so they can be applied one by one by orderof effectiveness. A central component in his book Software Security: Buildingthe security in [McG06] is the Risk Management Framework presentedthere. The risk management framework comes in a chapter before the seventouchpoints and should be performed both before and in parallel with thetouchpoints. A short overview of the Risk Management Framework (RMF)will be given and after that the seven touchpoints from McGraw will bepresented.
Risk Management Framework
The main idea behind the Risk Management Framework is to tie technicalrisks to business risks. This is to prevent what McGraw calls the techno-
17
gibberish problem, where business people don’t understand the consequencesof the risks involved. The problem with this is that business people don notsee the value of adding security to their software products if it is not tied toa business risk.
The RMF exists of five different stages listed below.
• Understand the business context
• Identify the business and technical risks
• Synthesize and prioritize the risks, producing a ranked set
• Define the risk mitigation strategy
• Carry out required fixes and validate that they are correct
The four last stages are iterated again and again as the risks changes andnew risks comes up. The RMF is supposed to be used continually duringthe development lifecycle to classify and mitigate risks. The RMF can alsohave an impact on which or how big degree the touchpoints are applied. TheRMF can also be used to track all the risks of a project, not just securityor privacy risks. It is also designed to be utilised in parallell with othersoftware development lifecycle activities. This is one of the most uniquecharacteristics with the seven touchpoints because it is heavily focused on riskmanagement, while for the other software security frameworks risk assessmentis just another activity on the same level as the others.
Compared to both SDL and CLASP the seven touchpoints and the RMFframework have a bigger focus on risk assessment, since the RMF frameworkis used extensively in the software development lifecycle.
Seven Software Security Touchpoints
The seven software security touchpoints is a set of best practices proposedin the book [McG06] and are different activities which can be used with anydevelopment methodology. In the book these activities are laid out in thetraditional waterfall lifecycle as shown in figure 2.3.
McGraw also claim that around 50% of vulnerabilities are due to imple-mentation bugs, and the other 50% are from architectural flaws, meaningdesign and architectural errors. Hence the first touchpoint is designed tocatch implementation bugs (Code Review) and the other is designed to catcharchitectural flaws (Architectural Risk analysis).
The touchpoints laid out in an order which makes the top of the list the mostimportant and the the order of importance declines down the list, however,all the touchpoints are quite important according to McGraw. By ranking
18
Figure 2.3: Seven touchpoints
the activities this way it is easy for a company to start a slow adoption of theactivities and begin with the activities that get the most return of investmentby choosing the activities on top.
• Code Review (Tools)
• Architectural Risk analysis
• Penetration testing
• Risk-based security tests
• Abuse cases
• Security Requirements
• Security operations
• *. External Analysis (Bonus touchpont)
As we can see from the touchpoints, many of the same techniques that werepresented in the other frameworks are here to. The interesting point hereis that McGraw only has seven touchpoints, whereas both Microsoft SDLand CLASP contains many more activities. The reasoning behind McGrawsdecision to only have seven, is that many frameworks with many activities canoften be overwhelming for an organization to adopt. These seven touchpointswhere carefully chosen as a means of getting software much more secure, butat the same time also not be overwhelming.
19
When we look at the seven touchpoints in the same phases as shown in figure2.3 we will start with abuse cases and security requirements. Abuse cases arevery similar to the misuse cases presented in the SDL and CLASP, and we willtherefore treat them as the same activity. Security requirements is also muchthe same as in both CLASP and SDL, however unlike SDL the touchpointssecurity requirements does not address privacy. The third activity shownhere which also touches several other phases of the development lifecycle isarchitectural risk analysis, just called risk analysis in figure 2.3. Architecturalrisk analysis, despite its name is quite similar with what the SDL and CLASPcall threat modeling. In his book McGraw argues that what Microsoft callsthreat modeling is what the academic security community calls architecturalrisk analysis. Also in the book[McG06] it is pointed out the similaritiesbetween the SDLs threat modeling and architectural risk analysis.
The next activitity is called risk-based security testing and it is divided intwo main parts, testing security functionality with normal functional testsand risk-based security testing based on attack patterns, risk analysis resultsand abuse cases. The latter part consists of making tests that test the systemfrom a security standpoint. These tests can be compared with the CLASPactivity identify, implement and perform security tests. Both CLASP andthe touchpoints have a big focus on security testing in comparison to SDLwhich focuses more on reviewing.
In the next phase during implementation the touchpoints mention the useof a static code analysier tool. The touchpoints argue that manual codereview is a difficult task which is both time consuming and give little returnof investment. An automated static code review tool on the other is cheapto implement and run and makes a great return of investment. Because ofthis the touchpoints only talks about source code review with a tool whichboth CLASP and SDL also does, but the latter two also mentions the useof manual code review, especially in SDL for the most critical parts of thesystem.
The next phase is the testing phase, here the touchpoints mentions the useof penetration testing. The touchpoints argue that the penetration testingshould be based of attack patterns, the abuse cases as well as the outputfrom the RMF framework. Penetration testing is also an activity in CLASP,but in the SDL it is mentioned as a optional activity that is good to performbut not necessary to be compliant with the SDLs recommendations. Thetouchpoints mentions the use of Fuzz testing but argues that it does not givea realistic view for how an attacker which are supposed to attack the systemwould behave, and therefore is less valuable that penetration testing.
Last in the phases of figure 2.3 is security operations. In the book [McG06]this point is not very well described. But it is essentially about the operatingenvironment which the software will be deployed on. This activity is quite
20
similar to the CLASP activity verify security attributes and test and verifythe security of it. Both of these activities describe that the operatingenvironment should be verified and secured and that for example properlogging should be in place to support forensics work.
The last point in figure 2.3 is the external analysis. This is drawn around allthe other activities and suggests that as many activities as possible shouldbe performed by an external entity to the project. The touchpoints mentionthat all the touchpoints should be reviewed by people outside the design andimplementation team. This is barely mentioned in neither CLASP nor theSDL. It is strictly not an activity but it is a good point and in the surveythat were created, many of the questions had a follow up question if anexternal entity performed the task.
Taxonomy of coding errors
McGraw also presents a taxonomy of coding errors and introduces somethinghe call seven kingdoms, which are different categories for coding errors. Thepoint of the taxonomy is to help programmers to be aware of the differentcoding errors they make. Since this taxonomy is not an activity in itself,it is not represented in the questionnaire for this report. However we haveincluded it here to make justice to McGraws book about software securityand we think it is an important tool in educating developers to write bettersoftware. This is similar with CLASPs taxonomy of vulnerabilities, howeverCLASP takes it a bit further and have a complete lexicon of vulnerabilitieswhile the touchpoints only have a taxonomy to help classify coding errorsand vulnerabilities.
Summary of Seven Touchpoints
The seven touchpoints are different from both Microsoft SDL and CLASP inthat there are only seven of them. Both the SDL and CLASP contains manymore activities which are to be adopted. However as the author McGrawpoints out, the touchpoints is a bare minimum, many other activities can beperformed for better security but the touchpoints show several activtitiesthat give a good return of investment.
Like CLASP the touchpoints are development methodology agnostic, andcould be used by both traditional waterfall, RUP or agile methodologies. Thisis unlike the SDL which have different versions for different methodologies.Also like CLASP the touchpoints can be slowly adopted, meaning thatonly a few of them are performed at first and as the implementer of thetouchpoints gets more comfortable with them is able to adopt more of them.
21
To support this the touchpoints are ranged in order of importance, or returnof investment making it easy to pick the most important ones.
Another interesting point about the touchpoints is their use of the riskmanagement framework, which is to be used continuously and in parallelwith the touchpoints. The seven touchpoints therefore has a much biggerfocus on risk management than both SDL and CLASP, which only have riskassessment as yet another activity and not as an essential part which is tobe used by all the other activities.
2.3.4 BSIMM
The Building Security in Maturity Model(BSIMM) [MCM09] is a maturitymodel for building more secure software. BSIMM tries to capture thecommon ground shared between different methodlogies, like the MicrosoftSDL, Owasp’s CLASP and McGraw’s Touchpoints. BSIMM is a collectionof good activities that can be used today. BSIMM is developed by lookingat which activities successful organizations have already done, and makingthat the basis for the BSIMM Software Security Framework. The authorspoint out that the creation of BSIMM was done by observations in the realworld and applying that. It is a maturity model since each actitity can bemeasured by 3 levels, and an organization adopting BSIMM can start withthe first level and work their way upward.
In the paper [MCM09] the authors refer to it both as BSIMM and BSIMM2.Since the framework was constructed using observations they first did 9interviews and created the BSIMM framework. After that they did 21 moreinterviews to verify the framework and did some minor changes to it andcalled it BSIMM2. In the report the term BSIMM is used for both BSIMMand BSIMM2.
Overview of the BSIMM framework
The BSIMM presents four main domains in the software development processand inside each domain are three main practices which are carried out. Itgoes even further and specifies a total of 109 activities, these activities areclassified inside one of the four domains and inside one of the three practicesfor that domain. It is called a maturity framework because each activity israted from level 1 to level 3 in order of maturity. The framework is also meantto be used by companies as they mature in the field of software security,meaning that as they get better they can adopt more of the framework. Theframework is summarized in table 2.2.
22
Table 2.2: The Software Security Framework(SFF)The Software Security Framework (SFF)
Governance Intelligence SSDL Touch-points
Deployment
Strategy andMetrics
Attack Models ArchitectureAnalysis
Penetration Test-ing
Compliance andPolicy
Security Fea-tures and Design
Code Review Software Envi-ronment
Training Standards andRequirements
Security Testing ConfigurationManagementand Vulnerabil-ity Management
The BSIMM framework have like McGraws book [McG06] a big focus onrisk management. Each of the main practices all have overall business goalsthat they try to fulfill in order to bring a reason for why an organizationshould do the activity. As the authors point out, by using risk assessmentone can see which activities are the most important to focus on. This is justlike the touchpoints which is no surprise since the author of the touchpointsis also one of the authors of BSIMM.
The authors of the framework emphasize that the framweork is to be used asa guide for an organizations own software security initiative. The activitiespresented can then be tailored to fit that organization. This is very differentfrom both CLASP, SDL and the touchpoints which is more focused on anindividual project. BSIMM covers an organization as a whole and have manyactivities that are meant to be shared between projects in the organization.
The four Domains of BSIMM
Since BSIMM contains 109 activities, they are broken down into 12 differentpractices in four domains. These practices are shown in table 2.2. this sectionwill walk through the practices and explain what BSIMM means with eachof them, it will not delve into the 109 smaller activities.
Governance domain, the strategy and metrics practice is about assigningroles and responsibilities, identifying software security goals, determiningbudgets and identifying metrics and gates. The strategy and metrics practiceis very similar to CLASPs monitor security metrics and institute awarenessprogram which is about assigning roles and responsibilities. The SDL alsohave activities that are covered by this domain, namely the define bugbar/quality gates. Compliance and policy in BSIMM is about setting aglobal organizational policy about software security, this is the same as
23
CLASPs identify global security policy activity. However the compliance andpolicy practice in BSIMM also focuses on testing and setting requirements forCOTS, this is activities that are mentioned in all the other three frameworks.The last practice is training, this practice is also well covered by the otherframeworks except touchpoints which mentions its importance, but have noformal activity for training of software developers or architects.
The next domain, intelligence covers the fields which the other frameworkshave placed in the requirements and design phase of a software lifecycle.This domain covers activities that are covered by all other frameworks aswell. The first practice is about attack models, meaning threat modelingfrom SDL or architectural analysis from the touchpoints, but also abusecases are covered here. In addition the attack models practice also coverstechnology specific attack patterns. The security features and design is a bitunique from the other frameworks. This practice is about making reusableand secure patterns and solutions that can be used across many projects.The topic of security patterns is mentioned and when we conduct differentsources we find much information about the topic, we will cover it in moredetail further down. The standards and requirements practice also coverssecurity requirements which all the other frameworks also have included.In addition it covers making standardization in the organization about forexample the usage of COTS. This is in contrast to the other frameworksbut since BSIMM is focused on an organization at its whole and not in aparticular project these differences are natural.
The next domain SSDL Touchpoints is very familiar to the other frameworksand all the activities covered by this domain is also covered by the otherframeworks. This practice involve architecture analysis, which means ar-chitectural risk analysis in the touchpoints or threat modeling in the SDL.Like the other three it also includes focus on code review with a static codeanalysis tools, however BSIMM also focuses on manual code review. Manualcode review is also covered by the SDL. The last practice is about securitytesting, this involves both black and white box testing techniques such as fuzztesting or risk based security testing. Security tests for the software beingdeveloped is also an activity that is shared with all the other frameworks.
The last domain is concerned about deployment. Penetration testing hasbeen moved here since it is best carried out in the operational environmentto test both the software as well as the operating environment. Penetrationtesting is covered by both the touchpoints and CLASP but as previouslymentioned is only an optional activity in the SDL. The software environmentis about configuring the environment in which the software will run. Thispractice covers the same activities as the touchpoints security operations, theSDLs release plan and CLASPs operational security guide. The last practiceof BSIMM is about fixing vulnerabilities after the software has been released,
24
this is much the same as the SDLs incident response plan and carrying itout. CLASP also have an activity covering this in its manage security issuedisclosure process activity.
BSIMM roles
The BSIMM framework as CLASP has a big focus on roles, and it especiallypoints out that the company applying the framework should have a SoftwareSecurity Group (SSG). They focused on the point that people in the SSGshould come from an software developer background, and then be thoughtsecurity, instead of getting network security people and learn them softwaredevelopment. This software security group is also to be used across projectsso that their work is shared among many projects and reusable. Both CLASPand SDL points out that there should be a security responsible or a groupon the project, this is not to be confused with the software security group inBSIMM which is project independent.
The most important roles for making a change in software security is howeverthe executive leadership. The authors point out that when they observedthe different companies all had some executive position regarding softwaresecurity. This is also in contrast to all the other software security frameworks,however since BSIMM is about an organization as a whole and not just asingle software project the lack of mentioning of software security in theexecutive level in the other frameworks is understandable. The touchpointsdo however, stress the importance of linking technical risks with businessrisks to engage executive management in software security.
The last role the BSIMM framework contains is everybody else. Builderswhich are developers, architects and their management falls into this rolein BSIMM. The SSG are supposed to work directly with the builders andhelping them perform different activities from the BSIMM framework. Asthe organization matures the SSG should train and learn the builders toperform the activities themselves and only help in certain special situationsand the like. The framework also lists testers, operations, administratorand executives and middle management in the everyone else category. Thetesters for example will be performing the security testing of applicationsand the operators need to learn of to securely deploy and host applications.
Summary BSIMM
The BSIMM framework is a comprehensive maturity framework for per-forming software security activities in an organization. It can be comparedwith CLASP in that both cover many activities and both allow a granular
25
adoption of the framework. Like McGraws touchpoints it roots itself in riskmanagement and uses risk management extensively in the framework.
BSIMM have a lot in common with all the other frameworks as can be seenwhen going through its 12 practices. There is one big differences however andthat is the focus BSIMM have on the organization as a whole while the otherframeworks only covers a single software project. This makes BSIMM quiteunique in that some of its activities only makes sense in an organizationalsize, especially about re usage of both patterns, middleware and knowledgeand share it across projects.
2.4 Comparison of the Frameworks
There are some activities that all of the big four frameworks have in common,these activities are sometimes described differently and there are some subtledifferences, but we find that the core essence is the same in all the activities.These activities are listed below.
• Source Code Review, both automatic and manually.
• Architectural Risk Analysis/Threat modeling
• misuse/abuse cases
• Security Requirements
• Security Operations/Environment
In the book [McG06] McGraw argues that what Microsoft calls Threatmodeling is what the rest of the security community calls risk analysis, hencethey are the same in our list of common activities. Since these activities arepresented in all the big frameworks, we consider them important activities inorder to get more secure software. Gary McGraw also points out that thesethree frameworks have a lot in common the the OWASP podcast[McG09] Inour questionnaire all of these activities are presented.
Source code review is an activity in all the frameworks, at least with automatictools. All the framework emphasize the usage of automated tools since itsvery cheap very effective at discovering vulnerabilities. Both the SDL andBSIMM also focuses on manual code review, while none of them goes into thespecifics of it, they both mentions that this should be carried out in additionto automatic scanning with a tool. Because of the lack of how manual codereview is to be conducted we have chosen to have two variants in the survey.One is peer-review, meaning that more than one person looks over the codebefore it is checked into a source code repository. Because of the lack ofguidelines for how to perform source code review in the frameworks we did
26
not have any follow up questions asking how it was performed but just askedin the survey if it was done and how often. The other type of code review ispair-programming, being a really popular type of code review with the rise ofagile methodology and Kent Becks extreme programming [Bec99]. Becauseof this these to in addition to static code analysis tools were chosen to be inthe survey.
The other main differences is the number of different activities in the differentframeworks. The seven touchpoints are only seven, but the book alsomentions the risk management framework which identify and rank risks. Inother frameworks risk management can be seen as another activity. Whenit comes to the number of activities, the seven touchpoints are the fewest,with only seven. BSIMM have a total of 12 main activities which in turn arebroken down into a total of 109 sub-activities. Since these sub-activities isjust a breakdown of the 12 main activities we will consider that the BSIMMonly has a total of 12 activities when compared to the others. Microsoft hasa bigger number of activities and also contains a set of optional activitieswhich can be done in addition to the normal activities presented in the SDL.The last one is CLASP which has 24 activities, however as we look into theseactivities we can see that some of the other frameworks have some biggeractivities that cover several of the CLASP ones.
As the authors of [DSB+09] points out there are some differences in thecoverage of the different frameworks. For example so does the touchpointstalk little about security training, whereas all the others have covered thetraining part. In the deployment phase the frameworks are also very different,the touchpoints only mentions that the InfoSec people should be involved,while the others have some activities in this phase.
When it comes to adoption there are several differences. While both CLASPand the touchpoints point out that they are process agnostic, it would bedifficult for a company to integrate them into their existing developmentmethodologies. However, CLASP contains a risk of mitigating an activityand the touchpoints are ranked in order of efficiency, making it easier tochoose which activities to adopt from them. The SDL has another approach,where it is defined for both traditional waterfall like development, they haveone AGILE variant and last they have one for line-of-business applications.The BSIMM framework is a maturity framework, which means that as thecompany can start out simple and then mature along with the use of theframework.
27
2.4.1 Survey Activities
All the activities that the frameworks had in common was included in thesurvey, chapter 4 contains a overview of all the activities in the survey. Wewill highlight some interesting activities which we included in the surveydespite the fact that it was only mentioned in one of the frameworks.
Fuzz Testing
Fuzz testing is an activity in the SDL and BSIMM. However it was includedin the survey since it has a strong role in the SDL, and in some ways seem tohave replaced penetration testing as a testing technique to be performed latein a waterfall lifecycle in the SDL. Because of this fuzz testing was includedin the survey because we were curious if fuzz testing dominates penetrationtesting or not in the industry, or if companies perform both at an equal level.
Security patterns
Security patterns in only an activity in the BSIMM framework. It is men-tioned in the book [McG06] but it is not one of the touchpoints. Since securitypatterns are reusable across projects we wanted to ask the participants inthe survey if they used them or not, this would be interesting since the usageof security patterns involves some sort of re usage but also that they caneasily apply well defined solutions to security problems. Security patterns isalso mentioned in several other academic papers about software security suchas [YWM08]. Another interesting fact is to see if the industry does someactivity which is not emphasized in the big frameworks. Because of this aquestion if the project use security patterns was included in the survey.
COTS
Review of COTS is mentioned in BSIMM and CLASP as separate activities.SDL and touchpoints both mention the review of COTS components. SinceCOTS review is mentioned in all the frameworks and very important in bothBSIMM and CLASP, we included both question if the participant performsreview of COTS as well as a follow up question asking how they did thisreview. This was also to see if much attention was payed to the usage ofCOTS and other frameworks when writing software.
28
2.5 Summary of Preliminary Studies
The first reserach question have been answered in this chapter. The smallliterature study gave an overview of what the biggest and most used frame-works for software security are and gave a foundation for how to find thestate of the practice done in the Norwegian industry. These big frameworksidentified here did all have a huge impact of which types of questions andalso how the survey was to be designed.
The four big frameworks found have a lot in common, but there was alsosome big differences. The BSIMM was the only framework that took a wholeorganization into account and not only a single software project. The otherthree frameworks focused on how to increase the security of one projectbeing developed. The level of guidance where also very different, the SDL isvery specific in how to perform many of its activities, while the other threemention guidelines but makes it up for the performer of the activity to figureout the details and tailor it to the specific project.
2.5.1 Research Question 1
The answer to research question one will be that the current state of the artconsists of four big frameworks which are complete frameworks that can beused during the whole development lifecycle of a project. These four are:
• Microsofts Security Development Lifecycle
• OWASP CLASP
• Seven Touchpoints
• BSIMM
These frameworks do not by any means cover all the state of the art in thefield of software security, but it gives us a good enough ground to make thesurvey and answer research question 2.
All these frameworks contains many activities that can be performed duringa development lifecycle, and many of these activities are also shared amongthem. This concludes the short literature study which found these as thefour biggest and most used frameworks at this point of writing. We wouldpoint out that a lot of research especially around individual activities exists,some of this research will be pointed out in chapter 4.
29
Chapter 3
Research Process
This chapter starts by listing the research question RQ2, then we discussthe choice of research method. Why we choose to use survey along withinterviews. After this we discuss how the survey was constructed and whywe made it the way we did both in terms of length and the reasoning behindthe questions we choose to have in the survey. We will also highlight thecompanies which participated in the survey.
3.1 Research Questions
The following research questions were stated during the start of the project.
• RQ2: How is software security done in practice in Norwegian industrycompared to the state of art research and theory
– RQ2.2: Does software security have the same focus in every stageof an applications lifecycle
How we dealt with RQ2 is discussed in the sections below.
3.2 Research Process
In research theory, there are two main strategies for collecting data, one isquantitative and one is qualitative according to Ringdal [Rin09]. We willdiscuss how we performed both of these method by the use of triangulationand which specific techniques we chose and why.
30
3.2.1 Quantitative Research Strategy
Quantitative can often use statistics to see trends, they are very structuraland friendly to statistical analysis. Another point of quantitative researchstrategies is that they are often much farther away from the research object,meaning that for example a survey is further away with little or no personalcontact with the respondents. It is both a benefit and concern with this, thebenefit is that the researcher is not likely to influence the results, the concernis that some information may not be surfaced because of the lack of personalcontact with the respondents. Since quantitative research is further away,it yields a lot of answers and much data compared to qualitative researchstrategies.
3.2.2 Qualitative Research Strategy
The other method is qualitative research. Qualitative research is done muchcloser to what we are studying. For example an interview will be quiteintimate between informer and respondent in contrast with a web-basedsurvey. Qualitative methods often tries to capture the relevant context inwhich the respondent is working in, this can be harder with an interviewbut for example a focus group or workshop might achieve that goal. Whena researcher is doing a qualitative method he is trying to understand therespondents situations, and get a hold of the key information which canexplain the respondents situation or actions, Ringdal [Rin09].
3.2.3 Triangulation
Triangulation is a technique where a researcher combines different methods.Meaning that the researcher uses both a qualitative method and a quantitativemethod [Rin09]. Often this is to get the benefits of both types of data andnot sacrifice any of the benefits from either of the strategies. There are twoways of performing triangulation, either the two methods, quantitative andqualitative, are equal or one method is just to support the other. Whena method is just support for another it is often done in a smaller scale tosupport the other activity. An example can be doing a survey before doinginterviews, the survey helps to identify the interesting individual which canbe interviewed and the researcher do not need to spend time interviewingnon-interesting subjects or many subjects which will likely answer the same.
In this study we did first construct a quantitative study and performed aqualitative later. If we did have the time we would have done qualitativestudies both before and after the quantitative. Performing qualitative studiesbefore a quantitative is to identify questions which can be very interesting
31
but which the quantitative would not cover and then include them in thequantitative study. The reason to do the qualitative in hindsight is to identifythings we might have overseen with the quantitative method.
For this study the qualitative research where used as a support method forthe quantitative. This was mainly due to restriction toward time and thefact that it is hard to ask companies to participate in two interview and onesurvey. We performed interviews after the survey, mostly to verify but alsoto try and discover things which the survey did not cover.
3.2.4 Research Strategy
There are five main types of research designs according to Ringdal [Rin09].These five will be presented in the table 3.2.4. These strategies describesboth qualitative and quantitative methods, and we present them to showwhich type of strategy we used to answer RQ2.
Research Design Strategies
Design Qualitative Quantitative
Experiments: possible, but seldomused
rare: Classical designfor cause analysis
Cross-sectionaldesign:
Very common: Personalinterview with few se-lected respondents.
Very common: Surveydone with a large popu-lation of participants
longitudinal: Common: Field obser-vation or personal inter-view on different pointsin time, focus on change.
Common: Two surveys,looks at prospective andretrospective surveys.
Case study: Very common: Field ob-servation or personal in-terview in a case
Common: Surveys per-formed in a case
Comparative: Common: Compare twoor more cases with aqualitative method
Common: Compare twoor more cases with aquantitative method.
Table 3.1: Research strategies, from Ringdal [Rin09]
These strategies are good for answering different research questions. RecallingRQ2 in this report, we are trying to figure out how the current state ofsoftware security is practiced in the industry in Norway. This means thatwe are interested in the current, and not in measuring something over time.Since longitudinal looks for changes over time, we can rule that strategy outsince it does not fit our research question. The research question also statesthat we are interested in the finding the general focus on software security,
32
not a particular case or firm. This two arguments make both Case study andComparative studies not fit our research strategy for answering the researchquestion. Experiments are more for doing cause analysis, RQ2 is not aboutany causes but more toward general tendencies in software security. Becauseof this the only one which really fits RQ2 is doing a Cross-sectional design.
For this reason we chose Cross-sectional design for our study. Below we willhighlight different methods of doing cross-sectional studies and describe whatwe chose and why.
3.2.5 Cross-sectional methods
In a lecture in the course TDT4180 Human Machine Interaction there werepresented several methods for doing observations, which will fit a cross-sectional study. These methods are from lecture 11 2009 from that course.The table was originally made for usability studies, but many of theseconcepts are in use in more general studies. It is included here as a rationalefor why we chose to do a survey and interviews.
Table 3.2: Cross sectional methods
Name Purpose Type ofdata
Good at Bad at
Survey Get answersto specificquestions
MostlyQuantita-tive
Easy toreach manyrespondents
We need toknow whatto ask
Interview Research.Get in-sight into asubject
Qualitative Give insightinto a prob-lem. Dialogwith the re-spondents.
Takes timeand do notshow actualwork context
Focusgroupsand Work-shops
Dialog witha group ofusers
Qualitative Get deepinto aproblem,involve therespondents
Need a lot ofresources
Fieldstudy
Study of theactual workcontext
Qualitative Understandthe re-spondentssituation
Takes time.Often need aclearance
As we can see in table 3.2 there are several methods which can be used whendoing a cross-sectional study. Focus groups and field studies unfortunately
33
both require a lot of resources. The biggest problem with doing focus groupsor workshops is the distance between us, the researchers, and the companieswhich participated in the study. Another obstacle is that for the companyit will take a lot more time to participate in a focus group compared to ainterview. The same arguments are true for field studies, but they oftenrequire some clearance to be allowed into the area where to perform the fieldstudy. Since a software project is something that involves a lot of activitiesa field study would take a lot of time to inspect all the different activitiesperformed in the project.
After the elimination of both focus groups and workshops, we only havesurveys and interview left. Since one is quantitative and one is qualitativewe wanted to to a triangulation with the use of both a quantitative andqualitative method we used both. The survey itself was web-based mainlybecause it is easier to handle, both in terms of getting the data digitallybut also in distributing it to the participants. The two methods chosen wastherefore web-based survey and interviews, the interviews where performedin hindsight of the survey to see if there was anything the survey did notanswer.
The research methods chosen to answer RQ2 was to do a quantitative methodwith a qualitative method as support. The reason for this was to get manyanswers in an easy way and be able to utilize statistical analysis on thedataset collected from the quantitative method. The qualitative was decidedto do both to support up the findings during the quantitative method andto find things that the quantitative method may had overlooked.
3.3 Survey Construction
The survey used in the project was made by taking a look at the four majorframeworks for software security discussed in Chapter 2 section 2.2. Thsurvey was first constructed on a white board, by grouping questions by thedifferent stages in a software development lifecycle. The different activitieswas then added from the different frameworks, starting with Microsoft’s SDLfor no special reason. When there where tasks which looked a lot like eachother, for example abuse cases and misuse cases they where consolidatedinto one question. This resulted in a long survey which where too long andit was then trimmed down to take about 15 minutes to complete. This wasto optimize the people actually answer the survey, more on that in section3.3.1. The final survey is described in chapter 4
34
3.3.1 A note about survey lengths
Before we present how the survey was constructed, a quick note about surveylengths is discussed. This is to rationalize why we decided to merge a lotof activities which did have some differences which we unfortunately had toignore during the making of the survey, it will serve as a justification for whywe had to make the survey as short as possible.
As they point out in [Cap09] when respondents answering a survey they willtire when the survey becomes long. When respondents tire the answers will beless accurate and they often just answer to be finished with the survey. Theyalso observed that respondents were more likely to skip optional questionsthe longer the survey was. Since the survey constructed in this report didhave a lot of follow-up questions, we were afraid that if the survey wasto long that respondents would answer so they did not get the follow-upquestions. As they point out in both [Cap09] and [Bog96] the response ratesand dropout rater where contradictory to what they call common sense. Thecommon sense often is that the longer a survey is fewer people will respondor drop out of the survey. However they point out that the longer the surveythere is a slight increase in the drop out rate, but not significantly higher.Since we wanted high quality answers, we followed the guidelines to keep thesurvey below the 20 minutes mark and tried to hit a 15 minutes mark forour questionnaire.
However, even tough response rates do not seem to drop significantly, westill needed to ask the different projects to answer the survey and they wouldlikely be more positive to a short survey than a long survey, after all we diduse their working time for the survey. In their paper [CCL01] Crawford etal. found that when the student where invited to take a survey that wastold to be shorter their acceptance rates where higher. They also point outthat those who got an email invitation, a personal login password and gotregular reminders where much more likely to take the survey. Unfortunatelythe survey constructed for this report is anonymous, and hence we couldnot send out email invitation because we did not know who our respondentswhere. However, as they pointed out the length is a factor, although notthe biggest it is certainly worth paying attention to, both with respect toresponse rates and quality of the answers.
Based on these findings a length of approximately 15 minutes where decidedand the inclusion of a progress bar where also included to further decreasedrop put rates as Crawford et al. point out[CCL01].
35
3.3.2 Initial Survey Construction
The initial design of the survey was done using a white board and makingall the different groups of questions that follow from the literature. Thedifferent questions groups initially proposed where:
• Practical Information
• Lifecycle
• Training
• Requirements/backlog
• Design and planning
• Implementation
• Testing and Verification
• Release
The question group ”Lifecycle” was removed and it’s questions where dividedamong ”Requirements/Backlog” and ”Design and Planning” question groups.The reason was that some of the questions where removed and others wheremerged together. This was done in order to keep the survey short, more onthat in section 3.3.3.
The initial survey design was done by systematically going trough the fourmain frameworks, starting with Microsoft’s SDL, then McGraw’s Touchpointsfollowed by CLASP and last was the BSIMM. The order is of no significance.Since many of these frameworks have many similar activities they wheremerged together in the survey to keep the length of the survey down. Thedown side of doing this is that there are some minor differences in theactivities that are similar which the survey won’t make any difference to.These differences where thought about but in the end we found that it wouldbe smarter to focus on getting the server shorter.
The initial survey contained over 60 questions, which was far to many. Weshow a refined version of the survey from the whiteboard i figure 3.1. Thefigure does not show all the follow up questions. As we can see the surveywas quite long and contained many questions and question groups. Thefigure also shows the different questions groups and which questions belongedto which group.
36
Figure 3.1: How the survey was originally made on a whiteboard. The arrowshow the order of the question groups
Practical Information
Industry the Project is in
Size of the project
Role In Project
Time working in the project
Time for the project
Time in the industry
Who is responsible for security
Security responsible, group or person?
Methodology
Software Lifecycle
Skale from 1-10 interest in security
Metrics
Bug bar
Classify security requirements
Security Training
Reading security news, blogs or podcasts
Security Certifications
Conferences
General interest in security
Requirements/Backlog
Security Requirement
Risk analysis
Abuse Cases
Security Policy
COTS
Design
Threat Modelling
Attack Surface Analysis
Security Patterns
Defence in depth
External analysis
Operating Environment
Identify data-resources
Identify network design
Attack patterns
Audit
Dataflow diagram
Reduce Attack Surface
Implementation
Patterns
Banned APIS
Static code analysis
Approved compiler flags
Test-driven
Peer-review
Guidelines/handbook
Continuos Integration
Testing
Pentest
Fuzzing
Dynamic Analysis
Attack Surface Review
Unit tests
System tests
Function tests
Release
Incident Response plan
Patching
Review Changes
Further Development
Time for fix
Test fix
3.3.3 Survey trimming
After the initial construction of the survey, it was noted by test participantsthat it was too long. As [Cap09] notes there is a significant drop in quality ofthe answers as the survey length increases, hence by having a shorter surveywe think we can avoid people just answering too quickly at the end of thesurvey.
The first things that where trimmed where the ”lifecycle” question groupwas removed and some of the questions where moved into the ”Require-ments/Backlog” and ”Design and Planning” question groups. Both the”Requirements/Backlog” group and ”Design and Planning” groups wherebecoming quite large, so many of the questions where removed from thosegroups. The questions regarding threat modeling and attack surface analysiswhere also shortened by just keeping them one place in the survey and alsocleaning them up. The first time these questions where made they asked alot of details regarding threat modeling and attack surface analysis. Theproblem is that there are many ways of doing threat modeling and manydifferent frameworks and modeling techniques used to do threat modeling.To prevent threat modeling from stealing to much of the surveys overall goalmany of these options where dropped.
37
The trimming also cut down the number of questions from over 60 to 38questions, including all the follow up questions. A complete overview ofall the questions which made it in the final survey is in appendix 4. Tomake the survey shorter we systematically did go through the survey andmerged activities together that was very familiar across the four frameworksas well as removing questions we thought less relevant. The testing phase forexample only included security related testing and did not ask any questionsabout general testing.
The final survey consisted of 7 questions groups and 38 questions in total.This gave the survey an approximate time to complete of 15 minutes whichwas the mark we set in subsection 3.3.1. The survey did also include manyyes and no questions, this will make it easier to perform statistical analysisof the results and see if there is any interesting correlations between differentactivities.
3.4 Projects to Study
This section will contain some information about the companies whichparticipated in the survey. Since the survey was anonomious we will notmention the companies with name, but rather talk about what types ofcompanies they where. In the survey we asked questions about the specificproject the responent was working on right now. So the size on the projectwill not reflect the size of the company. Big companies can have smallprojects and small companies can have big projects.
3.4.1 Consultant companies
There were several consultant companies who answered the survey andwas interviewed. All these consultant companies where either in norwayor it was their local office in norway which participated. The reason whymany consultant companies where asked to participate is that there arevery few companies in norway which produces commercial off the shelfsoftware. Because of this most of the software which is developed i Norwayis custom software. Many companies in Norway is quite small and thereforecannot afford to have in-house development teams and therefore often get aconsultant company to produce the custom software they need.
A total of 3 consultant companies participated in the survey. Two of themwere international and one where norwegian. The companies ranged innumber of employees from 100000, to 10000 for the two worldwide companiesand around 300 for the one in Norway. The participants in the survey whereall from the norwegian offices for all the three consultant companies.
38
3.4.2 Non-consultant companies
There where two non-consultant companies which participated in the survey.One of these companies where part of a large group, but in this survey onlythe Norwegian office did participate since the survey is about the industryin Norway. The second company is Norwegian and sprung out from theNorwegian University of Science and Technology. These two companies willrepresent the non-consultant companies in Norway. We tried to contactmany other companies, but either we did not get a reply to our email or theysimply did not have the time or resources to participate in the survey.
3.5 Summary of research process
We started this chapter by arguing which research strategy and methods werebest to answer research question RQ2. We concluded that a triangulationof both quantitative and qualitative methods would prove the best. Thestrategy chosen was a cross-sectional study since we wanted the state of thepractice as it is right now. The specific methods chosen were web-basedsurvey for the quantitative data and interviews for qualitative data. Thesewere mainly chosen because of the few resources they required compared tothe other methods.
The survey was constructed by first taking all the activities from the theoryand add them as questions to the survey. Since the survey then became tolong we did a trimming process which brought the survey down to about15 minutes of completion time. At last we had a quick overview of theparticipants in both the survey and interviews. Unfortunately not all thecompanies from the survey participants had time to perform interviewsafterwards.
39
Chapter 4
Activities in the Survey
This chapter will walk through the survey. The survey contained severalquestion groups which are the sections here. We will discuss the differentactivities that we asked about in the survey bot in terms of the four frame-works, but also look at other litterature about the same topics. This willnot be a complete overview over all the questions in the survey, but morediscuss the theory used in the survey. For a complete overview of the surveysee appendix D
Since the survey could not contain all the activities from all the big fourframeworks, we had to take out only the most important activities and covertheir topics here.
4.1 Practical Information
Since the survey asks a particiant about the project he is working on rightnow, we start the survey by collecting some practical information about theproject itself.
The first question group in the survey is some practical information aboutthe participant which is answering the survey. This has nothing to do withsecurity but we are interested in how big the projects is, both in terms ofpeople involved as well as timeframe of the project. We also ask who isresponsible for the security in the project and if it is a customer or in thecase of a consultant company if it is the consultant company itself. Thisinformation is not from any framework or other sources but is some practicalinformation which can be interesting when analyzing the results, for examplesee if there is a connection between size of the projects and the activitiesthey perform.
40
4.2 Training/Education
The second section is about training and education. Strictly not a stagein a waterfall process, it is the start of the Microsoft SDL and many otherframeworks mentions training and education before they dwelve into thedifferent activities, hence it was the first question group in the survey.
Microsofts SDL explicitly points out that every member of a software devel-opment team should have received proper training. They list both basic andadvanced topics software developers should know about. CLASP mentionsthat an organization should establish a security awareness program and thatprogram should ensure proper training. McGraws seven touchpoints onlymentions that the software developers should have proper security training.One of the main points in BSIMM is training. BSIMM lists several pointsunder training, but most of them are like CLASP, they specify that thereshould be an awareness and how the Software Security Group (which isessential in BSIMM) should have office hours, and do training, but nothingas specific as Microsofts SDL.
As we can see all the major frameworks mentions training, altough the SevenTouchpoints only points it out a little. Since proper training seems to beimportant we have had some questions regarding security in the survey.These questions do not go as deep as the Microsoft SDL does by mentioningspesific tecniques, but rather asks how interested the participant is in securityand asks if the person has any relevant certfications. Unfortunately thereseems to be few security certifications which goes into the topic of softwaresecurity, some programming certifitcations have security build onto themand the certfication about security is mostly from a network standpoint.Despite this we have included many certifications about network security inthe survey.
4.3 Requirements
This section covers the requirements section of a waterfall model.
McGraw’s seven touchpoints have three of it’s touchpoints touching thisstage of the development cycle. The touchpoints are abuse cases, securityrequirements and risk analysis. Security requirements are also mentioned inMicrosofts SDL. The SDL also mentions security and privacy risk assessmentand the establishment of quality gates and bug bars. These are quality levelswhich should be set during this stage and never relaxed.
The last two frameworks, BSIMM and CLASP both have activities regardingrequirements. BSIMM mentions that the Software Security group should
41
have several activities which establishes security requirements both for code,features and compliance. CLASP also mentions variations of security re-quirements and abuse cases. CLASP calls abuse cases for misuse cases andthe activity it defines will detail the misuse cases. We will go trough bothsecurity requirements and abuse cases in a bit more detail below since theyare mentioned in almost the frameworks as well as other papers.
4.3.1 Abuse Cases
Abuse cases have existed for a long time, and they can often be used toelicit security requirements as Guttorm and Opdahl point out in their paperfrom 2000 [SO00]. Abuse cases are sometimes called misuse cases and theredoes not seem to be any difference between the terms, for example McGraw,Hope and Anton mix these terms together in their paper from 2004 [HMA04].Because of this we have used both terms in the survey for the same question.
Abuse cases are useful for getting into the mindset of an attack as Hope,Anton and McGraw points out [HMA04]. They will help companies thinkmore as attackers do and can be a useful tool. Guttorm and Opdahl discussin their paper [SO00] about using abuse cases as a basis for making securityrequirements. They extends normal use cases to include misuse cases and usethem to elicit security requirements. They discuss the strengths of misusecases and one them being that misuse cases are very informal and alsoprovides focus on security before the design phase.
4.3.2 Security Requirements
We use the excellent paper by Moffet, Haley and Nuseibeh [MHN04] whichbrings together the field of requirements engineering and security engineer-ing to define what security requirements are. In their paper they makea practical and useful definition of security requirements and places themin between security goals and security functions in terms of abstraction.Security requirements are mentioned in all the big frameworks, but very fewof them describes in great detail what security requirements exactly mean.Microsofts SDL talks about Security Requirements as being more qualitydriven activities for software development which are required, and example is”Identify the team or individual that is responsible for tracking and managingsecurity for the product.”. This is in big contrast to for example the seventouchpoints, CLASP and BSIMM which use the term more aligned withMoffet, Haley and Nuseibeh.
Since only Microsoft differ from the other three big frameworks securityrequirements in the survey are more along the lines of Moffet, Haley and
42
Nuseibeh and hence also CLASP, Touchpoints and BSIMM. Microsoftsdefinition is by no standard wrong, it just means something else than theother definitions.
4.4 Design and Planning
The next stage in a traditional waterfall process would be the design andplanning stage. This is also the next block of questions in the survey andmostly contains questions regarding architectural risk analysis or threatmodeling, we will discuss the similarities between these two terms shortly.
In the design and planning phase all the different frameworks have manyactivities, this is probably since this is a phase where it is easiest and leastexpensive to find and fix architectural flaws1 and bugs. As Tassey [Tas02]argues, the cost of fixing a software defect, let it be security or other type ofdefect the cost increases exponentially along with the development lifecycle.However Kent Beck argues in his book [Bec99] that the cost of fixing softwaredefects is about constant since the feedback loop is so short. However seeingthat many systems still have a QA phase with heavy testing a combination ofboth the old waterfall cost and the agile constant cost should be considered.In this report we assume that the cost will increase further in the developmentprocess, but we will not make any claims by how much.
Because the cost of fixing defects increase as the software gets more andmore developed, it can be a good idea to have some threat modeling orarchitectural risk analysis done early in the planning stages, regardless ofdevelopment methodology.
4.4.1 Architectural Risk Analysis
Before we delve into the topic a note about Architectural Risk Analysisversus Threat Modeling. In his book about software security McGraw arguesthat what Microsoft and many others in the industry calls threat modelingis what academics calls Architectural Risk Analysis. In this report we try tostick with the term Architectural Risk Analysis, but in the survey we usedthe term threat modeling because it seems to be more widespread amongindustry people in Norway.
Architectural Risk Analysis is mentioned in all the big frameworks, but as westated in the beginning, by somewhat different terms and methods. BSIMMand the seven touchpoints mean the same when they mention Architectural
1Architectural Flaws are design errors which McGraw introduces in his book aboutSoftware Security [McG06].
43
Risk Analysis, probably because McGraw has a finger in both frameworks.Microsoft’s SDL and CLASP however both differ a bit in what they meanwith Architectural Risk Analysis, or threat modeling as Microsoft calls it.CLASP use the activity
Perform security analysis of system requirements and design(threat modeling)
The activity description looks quite familiar to both the touchpoints andBSIMM, but contains a lot less details. The SDL uses threat modeling asthe term for architectural risk analysis. However all these methods in all theframeworks collect the assets in the system and uses the assets to construct amodel of all the threats and ranks them by risk. However they vary alot withexcatly how you identify the assets and identify the risks, the core essence isthe same.
4.4.2 Attack Surface
When it comes to the attack surface both the Microsoft SDL and CLASPmentions that you should identify and reduce the attack surface as muchas possible. They both also mentions that by identifying the attack surfaceyou can do a architectural risk analysis on the different parts of the attacksurface.
On the other side, the touchpoints nor BSIMM mentions anything aboutattack surface, but they do mention threat identification. Threat identifcationis a part of the architectural risk analysis. In Microsofts SDL the attacksurface identification is also a part of the threat modeling. Altough sincethey are seen as separate activities in CLASP and sub-activities under threatmodeling in the SDL, we have included a separate question about attacksurface identification in the survey in addition to the architectural riskanalysis question.
4.4.3 Security Patterns
Security patterns is a classical design pattern designed to fulfill a securitygoal. The point of security patterns is to contain domain independent,time-tested security knowledge and expertise. Security patterns containsall this knowledge in a reusable format so that non-security experts canutilize the patterns in their design. Yskout Koen et al did a technical reportin 2006 [YHSJ06] where they had collected and reviewed many securitypatterns. There have been many other report and papers at both genericsecurity patterns as well are more speicific patterns such as secuiryt patternsregarding web applications, such as [KE02] from Keinzle et al.
44
Two of the four frameworks we are comparing mentions security patterns.BSIMM has a category called security features and design, where one activityis that the software security group should focus on making secure patternspublishing them for the organization. The SDL mentions security patternsbriefly, and contains links to Microsofts Patterns and Practices 2. Thetouchpoints and CLASP do not mention anything about security patternsdirectly but they both mentions the use of industry best practices, and wethink security patterns should be one of the industries best practices.
4.5 Implementation
All the big frameworks does contains automatic source code review. Manyalso mentions manual review of all or parts of the source code. Sourcecode review have existed for a long time, which the old Fagan methods[Fag76, Fag86] is a good example of. Today much of the software developedis using an agile methodology and utilizes many principles from ExtremeProgramming[Bec99], such as pair programming. By doing pair-programmingmore than one person has already seen the source code and this is often usedas a substitute and faster way of doing code review today.
The advances in tools to help with source code review, called static codeanalyzers. McGraw makes a big point in his book [McG06] about using toolswhen doing code review, simply because manual code review takes too muchtime. Microsofts SDL also recommends using static code analyzing toolswhen developing code. The SDL also contains a set of banned and dangerousAPI which the developers should avoid when doing coding. The SDL incontrast to the other three frameworks contains lot of specific knowledge inthis area, about which API’s to use, which not to use, how to use them andso on. BSIMM goes the other way and states that the Software SecurityGroup is responsible for creating these lists, and choosing code review tools,banned API’s and so on. CLASP and the touchpoints take the same view asBSIMM and that is that the people developing the software as responsibleto create their own lists and choose their own tools.
All the frameworks have a huge impact on implementation specific activities.All of them contains code review, and many mentions developers handbooks,list of banned API’s and so on. The SDL is unique here in that it definesand lists many tools and documentation to be used for this process whilethe others leave that up to the one using their framework to enhance thesoftware security of their project.
2http://msdn.microsoft.com/en-us/practices
45
4.6 Testing and Verification
Penetration testing is a big topic here, all the frameworks mentions italthough the SDL only mentions it as an optional activity. Microsoft onthe other hand have a big focus on Fuzz-testing which is something thatis not mentioned in any of the other frameworks. Microsoft also have anactivity with dynamic analysis, which means to observe the program andhow it behaves during runtime. This dynamic analysis is also unique to theSDL compared to the others. The last type of testing we will present here issecurity tests, which aims at testing security functionality. This is somethingthe all the frameworks mentions, however in different quantity. Microsoftdoes as mentioned earlier have a bigger focus on fuzz-testing, especiallyprotocols. The touchpoints and BSIMM all have security testing, but theyare ranked less important than penetration testing. CLASP is quite vaguein what is means by security testing, since there is no definition or exampleonly an activity that says to make security tests.
4.6.1 Penetration testing
Penetration testing is nothing new, and is perhaps one of the most commontests for security vulnerabilities. Even though it is quite old it is still reallyvaluable because as McGraw says in his book [McG06], security testing setsyou in the mindset of an attacker. Since penetration testing has been aroundfor a long time, there are many guides to help perform a penetration test,one quite common is the OWASP testing guide[Gui08]. Penetration testingis also mentioned in NIST Special Publication 800-115. Since penetrationtesting is universially menitioned in all the frameworks, it received a greatdeal of follow up questions in the survey.
4.6.2 Fuzz-testing
Fuzz-testing is certainly nothing new, a website containing many paperson fuzz-testing of different operating systems http://pages.cs.wisc.edu/
~bart/fuzz/ dates back to 1988. The most interesting thing we found wasthe big focus Microsofts SDL has on fuzz-testing. BSIMM also mentionsfuzzing but only of network protocols and certain APIs. The SDL hasa big focus on fuzz-testing and almost ignores penetration testing, whichtouchpoints and BSIMM focus a lot on.
46
4.7 Release and Maintenance
The last category of questions we had in the survey was about release andmaintenance of the software to be produced. All the frameworks have differentparts here and again the Microsoft SDL is very specific while CLASP is a bitmore vague and seems to leave much up to the users of the framework. Allexcept the touchpoints have some sort of an incident response plan whichcan be used to track and rank vulnerabilities and bugs which are found afterrelease.
4.7.1 Security Operations
This topic is about security configuration and requirements for the operatingenvironment for the software which are produced. All the frameworks havepoints in that the operating environment should be secure. CLASP has somespecific activities such as ”specify database security configuration”, whilethe touchpoints mentions that security operations is quite important, butnothing as specific as CLASP. BSIMM has one of it’s main activities as beingSoftware Environment, which discusses how the environment the software tobe released in should be configured. The Software Security Group plays abig role in deploying the software in the BSIMM framework.
Microsofts SDL is the one with the fewest requirements on the operationsenvironment the software to be released will be in. This can be argued thatthe SDL is strictly about how to produce secure software and that securingthe operations environment is another topic.
4.8 Summary of Survey Activities
All the questions groups and the major topics they cover have been coveredhere. As we can see there are also much research done outside of theseframeworks, and many of these frameworks in turn uses the latest researchwhere appropriate. The survey therefore mirrors the current research andmost used activities in software security and will give us a good representationof how many of these activities the industry actually perform. For a completeoverview of all the questions see appendix D.
47
Chapter 5
Results
This chapter presents the results from both the survey and the interviews.The survey results will be looked at initially here and we will take a deeperlook and analyze some of the results in chapter 6 We will start this chapterby going through the interesting results from the survey. After this we willwalk through the interviews and compare the results from the interviewswith the survey results.
5.1 Survey results
In table A.1 are the results from the survey aggregated. There were a totalof 22 answers with 19 complete and 3 incomplete answers. In table A.1 areall the 22 answers and the ones that are incomplete are shown in the ”Notcompleted or not displayed” measurement. Not displayed can also mean thatit was a follow up question which was not shown because the user did notanswer what would trigger the right condition for the follow up question tobe shown. Below we will discuss the interesting results from the survey.
5.2 Discussion of the survey results
The results from the survey show that only some projects have a focus onsoftware security and that the trend seems to be a bit more binary, eitherthey have a focus and does many activities or they do not do any or veryfew activities. However there where many smaller projects that participatedin the survey, 8 of the projects was in the 1-5 size category. With that fewteam members the pressure to make formal activities with regards to securityprobably diminishes.
48
5.2.1 Project sizes and time spans
When we look at the results from the project sizes and time spans of theprojects, they vary much in terms of time spans, while there are manyprojects which have a rather small development team. These results arequite important when we take a look at the other results. We need to keep inmind that often smaller projects over a short timespan will have fewer formalactivities and processes compared to bigger projects in terms of both sizeand time. One weakness of the survey results is the lack of bigger projectsin terms of size. While we have many respondents from smaller projects wehave only a few which are over 40 persons, and just a handfull that are over20. These make it impossible to draw any conclusions regarding softwaresecurity and size of the project.
Figure 5.1: Project size and time span
5.2.2 Other practical information about the participants
When we look at the question regarding who is responsible for the securityof the project the participants was being asked about. We found that oneoption we missed was developers as five of the participants had writtendevelopers or similar as their answer to this question. Another interestingresult is that 16 of the participants did have the role as developer. Fivehad a role as project manager, and some even had both. Also most of theparticipants had worked on the project for 1-2 years or less. For all thepractical information, see table A.1
5.2.3 Evaluation of COTS
Very few of the participants did any review of COTS, only 4 out of 20said they performed review of COTS. One interesting point is the followup questions which asked the participants to check for what they did whenreviewing COTS. All the four participants did a check with community, alook into NVD or other vulnerability databases and everyone did internal
49
testing of the COTS. None of the participants did any external testing nortrusted the vendors testing of the component. One interesting fact is thatthe participants did not trust any external entity with the testing of thecomponent, but the community seemed to be trusted, perhaps since it isan independent third party which have no interest in promoting the COTSunlike a vendor.
5.2.4 Results of many yes and no answers
In figure 5.2 we have taken some of the yes and no quesions from the surveyto highlight that about half of the participants did perform the differentactivities and half of them did not. In chapter 6 we will show that most ofthe time it is the same participants which performs the activities and oftenthe same participants who do not perform them. In table A.1 we can seeeven more of these types of answers to many of the yes and no questions.This supports one of the major results that help us answer research questionsRQ2. Namely that the software security done in practice is divided suchthat about half of the participants did perform software security activites,while the rest did not.
Figure 5.2: Penetration Testing, Risk Assessment, Attack Surface analysisand Security Requirements results
5.2.5 Threat modeling
One interesting fact is that very few of the participants performed threatmodeling of the architect, while many performed attack surface analysis anda risk assessment. We have one theory about this and that is that since most
50
of the development projects did follow an agile approach, see the table A.1for the different methodologies used. In an agile approach one of the mainprinciples is to refactor mercilessly and therefore the architecture is subjectto change often, which makes is difficult to perform a threat model on thearchitecture since it often changes.
We would like to clarify that the reason for little threat modeling mentionedabove is just a theory and we currently have no data to provide an accurateanswer to this question. This result will therefore require further researchbefore an answer can be provided.
5.2.6 The activities that almost none did
There was also some activities which some of the big frameworks promotedwhich had little to no presens in the survey results. On of these activitiesis fuzz testing as mention in subsection 6.2.4. There was only 3 out of 19participants that performed fuzz-testing. None of the participants had anybanned APIs, suggesting that this activitity is not in use in practise. Ifwe compare this with the frameworks it was mostly mentions in MicrosoftsSDL and also mostly for the C and C++ languages. Considering thatthese languages are not much used in modern web-applications, perhaps thisactivity is becoming obsolete when developing web-in palapplications moremodern programming languages.
The last activity which very few performed was dynamic analysis. Only 2out of 19 participants performed this activity. This suggests that perhaps itis too much expense to do a dynamic analysis of the application during runtime, or that it is just not a popular or cost effective method. We currentlydo not have enough data to identify the reason why dynamic analysis was sounpopular so we can only make guesses.
5.3 Results validity
One concern we had with the survey was that we would only get participantswho already was interested in security. Regarding many of the responseswhich mostly did not include many security activities, we find is safe toassume that this was not the case and that many of the participants didnot have a particular interest in security, compared to a random selection.Since the survey is anonymous we can not see which companies or howmany responses from each company we have received. Because of this wecan only hope that the data is randomly distributed among the companieswhich participated. This is hardly the reality and should be taken intoconsideration by any reader of this report. The analysis and findings here
51
make the assumption that the data collected were distributed evenly amongthe participants.
One quite interesting finding is that nearly all the participants had a planfor handling bugs and problems after release and had an easy way to patchthe system. This points toward a trend that security is often though of as areactive process, rather than proactive, in the analysis chapter we will takea closer look at correlations between different activities.
5.4 Interview results
The results from the interview are presented only to supplement the survey.Hence they are only included in the appendix in their raw form. We havetaken the interview results into account when reading the survey results andcheck that they match. The interview will therefore mostly be used to matchthe results from the survey, in chapter C we have gone through the phasesfrom the survey and compared them with the results from the interviews.
The interviews was originally supposed to be done at least one at eachcompany which participated in the survey, however due to difficulty for theparticipants we only managed to get interview with two of the companies.Of the two we had one company which had time to participate in 4 interview,while the other company only had time for one company. We would like topoint this out since the interview results will be heavily influenced by onecompany but it still matches the survey results remarkably well.
The results from the interview match the survey in most cases, we often seethe same results with only small changes this is mostly due to the interviewbeing only five compared to the 19 full results from the survey. We had oneinteresting discovery during the interviews, and that was that one of thecompanies in the interviews had their own security review process. We willhighlight this process in section 5.5.1.
5.5 Summary of the interview results
From the interviews in chapter C we can see that it matches the findings inthe survey pretty well. There were some findings in the interviews that didnot match the survey this can have several causes. One was that not all thecompanies which participated in the survey was present in the interviews.Another factor is that the interview where heavily influenced by one company,whereas the survey probably was more evenly distributed.
52
One thing we discovered during the interview was that one company oftenhad a security review process which we will describe in more detail below.
5.5.1 Security review process at one company
The company which had the security review had a security group whichfocused on writing software more secure from scratch. This group sometimesperformed a security review of a project that was going on. This is a shortsummary of what such a security review involves.
The security review goes through the functional solutions in the software. Italso review the architecture with both people from the security group anddevelopers of the software. They identify the assets of the application anddoes a quick risk analysis and identifies the biggest threats to the application.They then for the easy to perform attacks performs them on the system, thisis mostly done manually and with little automation. The whole process isdone white box, since it also consists of a code review of critical securityfunctionality, often in pair with both developers and one from the secuitygroup.
The security review produces a report which takes into account both whathas been found, which code has been reviewed but also code that has not beenreviewed. They emphasize that they include what has not been reviewed inorder to not give the developers a false sense of security.
By the end of the review all the findings are presented and dicussed togetherwith the developers, this both helps as training of the developers as well asimproving the security of the software. This have some similarities towardwhat Microsofts SDL calls a security push in the verification phase.
53
Chapter 6
Analysis
This chapter will start by discussing some of the general trends found inthe survey, these trends will be further explored in a section which look forstatistical correlation between different activities in the survey.
6.1 General trends in the survey
When we look at the results from the survey, we can observe that in generalthe people who participated was developers and project managers. Sincethis was a field with check boxes two were both a developer and projectmanager. There was several others also, but the majority was developers.This is something to take into consideration when reading this analysis.
By simply skimming the different results, it seems like there is a binaryapproach to software security, either they do many activities or they almostdo none. Further down we will look into this trend by statistical analysis.
We can see from the findings from the questions about security requirementsin figure 6.1 and Risk assessment in figure 6.2 that these are almost evenlydivided. If we check the results from each participants in the survey, we cansee that almost everybody who had security requirements also did a riskassessment and the opposite In subsection 6.2 we will dig deeper into thesekinds of correlations.
Another interesting finding is that almost every respondent in the survey didhave a way of patching the system they were developing in case any seriousvulnerabilities was found. This is the more classical reactive approach tosecurity, which can be argued to be inferior to proactive security in someways[CKMR03]. However reactive security is better than none security andas the discussions in [CKMR03] says, reactive security is in some places the
54
Figure 6.1: Answers to: Do you have specific security requirements?
Figure 6.2: Answers to: Do you perform a risk assessment to determine thebiggest risks for the software?
55
right solution. It is hard to draw any conclusions based on this but it mightseem that at least everybody is thinking reactively about security.
6.2 Correlation between different activities
To measure a correlation between two activities we have used Pearsons R[Rin09]. This number gives us an indication if there is a correlation betweentwo different values, meaning that they have an effect on each other. Inthe survey we had many yes and no answers if the respondents performeddifferent activities. The kind of correlations we will look into is the correlationbetween these types of activities. These correlations will indicate if thereis any statistical correlation between these activities, meaning that of aparticipant answered yes on one question there is a higher chance he willalso say yes on the second activity. Before we presents the numbers, we willexplain what Pearson product-moment correlation coefficient is and how wecan interpret the results.
6.2.1 Pearson product-moment correlation coefficient
Pearson product-moment correlation coefficient sometimes called Pearsons rcan be expressed mathematically in many ways. One of the most popularforms is shown below.
r =cov(X,Y )
sXsY=
∑(Xi − X)(Yi − Y )
sxsy(n− 1)(6.1)
where
sx =
√∑(Xi − X)2
n− 1sy =
√∑(Yi − Y )2
n− 1
The equation (6.1) gives us a coefficient between -1 and 1 which tells ushow much the values correlate between each other. Pearsons r determinesif the values resembles a linear dependenc of each other. The value goingfrom -1 to 1 can be interpreted in different ways. A value of 1 means aperfect positive linear correlation, while a -1 means a perfect negative linearcorrelation. A value of 0 means that there is no correlation at all. This isbest illustrated with a figure 6.3
56
Figure 6.3: Correlation of different value of r
When doing a statistical analysis with Pearsons r we must do construct aconfidence interval around r that has a given chance of containing p, where pis the true correlation coefficient in the data set. We make a null-hypothesisthat the value of p is 0. The null-hypothesis will be reject if value of thetest statistics is sufficiently large or sufficiently small. We use a two-sidedtest which means that the value that either of the regions of the confidenceinterval is used. This is shown in figure 6.4. The null-hypothesis is tested bymaking a random distribution of the dataset and compare those results tothe measured values of the data set.
Figure 6.4: Confidence interval showing both tails
6.2.2 The usage of Pearsons r
In this report, we have used Pearsons r to show linear correlation betweenactivities. Since Pearsons r uses a plot of numerical value we had to translateour answers from the survey into numerical values which could be used to
57
measure Pearsons r between variables. From the survey we took all the yesand no questions and translated the yes answers to 1 and no answers to 0.This makes it easy to calculate Pearsons r and only 1 and 0 makes onlylinear correlation relevant, hence the usage of Pearsons r and not some othercorrelation coefficients.
To interpret the results from Pearson r, we use the following table 6.1 todetermine the strength of the correlation. This gives us a solid groundfor how to interpret the results we get from the correlation analysis of thedifferent activities.
Table 6.1: Strength of Pearsons rCorrelation Negative Positive
None -0.09 to 0.0 0.0 to 0.09
Small -0.3 to -0.1 0.1 to 0.3
Medium -0.5 to -0.3 0.3 to 0.5
Strong -1.0 to -0.5 0.5 to 1.0
For our analysis, we generally have to hypothesis, one which we call H0 andone which we call H1.
• H0: The true correlation coefficient p between two activities is equalto 0, based on the sample correlation coefficient r.
• H1: The true correlation coefficient p between two activities is notequal to 0, based on the sample correlation coefficient r.
We use a two-sided test and accept a 95% confidence interval, meaning that2.5% is taken on either side. This is shown further down in table B.1 whereone asterisk mean a 95% confidence and two asterisks mean 99% confidence.
58
6.2.3 Correlations
Table B.1 shows all the different yes and no answers and the calculatedvalue of pearsons r in the intersection between two activities. As we can seethere are many both strong and medium correlations in the table. Theseresults show that when a participant in the survey is doing one activity,such as Security Requirements, they are also very likely to also do an attacksurface analysis. This points towards a more binary relationship between theactivities, meaning that either a participant does many of the activities or theparticipant does none. Below we will point out some of the more interestingcorrelations from the survey and look at these in regards to research questionsRQ2.
However, as we can see many of these activities in table B.1 does not containsan asterisk or two (* or **). This means that there is not a statisticalsignificance in the correlation and we cannot trust the correlation coefficient.An example is between the Peer Review and the Evaluation of cots, eventough Pearsons r is quite large and would suggest a medium correlation, thesamples are two small and it could very well be luck that these two valueshave a correlation, hence we cannot state that there is a statistical significantcorrelation between these to activities.
6.2.4 Fuzz testing, odd results
We have one quite odd results which is fuzz-testing. Some places it is actuallynegative correlated to other activities, meaning that if they perform theother activity they are less likely to perform fuzz-testing. However since veryfew performed fuzz-testing this number is not significant, none of the fuzztests are significant at the 95% level. The fuzz test results should thereforebe ignored since they are likely just some statistical oddity due to the smalldataset collected for it.
6.2.5 Penetration testing, an entry gate
Another interesting result is penetration testing. Penetration testing ispositive correlated to all the activities except fuzz testing, dynamic analysis,attack surface analysis, patching and release plan. This gives an indicationthat if a company performs penetration testing, they are also very likely toperform many of the other activities. When we look at the correlation withattack surface analysis the results are significant at a 0.081 level, meaningthat the results are significant at a 90% confidence interval. However sincewe used 95% in our analysis we have to assume that there is no significantrelationship, but we wanted to point out that in a 90% level, it would be a
59
medium correlation. These findings makes it looks like penetration testingis an entry gate into performing other activities. This also suggests thatpenetration testing is one of the activities that almost everyone that have afocus on software security let the system undergo a penetration test.
Penetration testing is interesting when compared to the four big frameworksused to produce this survey. All the big frameworks mentions penetrationtesting except for CLASP, which has less focus on traditional testing andmore about the development itself. This suggests that it is an quite importantactivitiy. In the touchpoints it is number three in the priorities list, andBSIMM also points it as one of the most important activities. We also wantto highlight one of the companies from the interviews who did a securityreview, which included penetration testing. Penetration testing seems to beone of the most important and dominant activities except reactive patchingand fixing in the industry in Norway.
6.2.6 Reactive security, the most dominant
The third result we want to highlight from table B.1 is the operating en-vironment requirements, the ability to patch and a release plan. The lasttwo have a positive correlation with almost all the other activities, howeverthey are not statistically significant. This is because since almost everybodyin the survey did answer yes to these two there are very few no answerswhich can be correlated to the other activities no answers. This means thatit is very likely that these positive correlations are due to statistical entropyand would not pass our 95% significance level. The same conclusion goesfor operating environment requirements but it has some no answers and iscorrelated at an acceptable significance level compared to some activities.
These findings suggests that the traditional reactive approach to securityis still dominant. However we would like to point out that even proactivesecurity is quite important it is almost impossible to foresee every threatto a system, and companies needs to react to new threats that can happento a system. In light of RQ2 this is the most important activitiy that allthe norwegian companies does. It is highlighted in all the frameworks to adifferent degree as discussed in section 4.7. In summary we can conclude thatthis is the most important activitiy which everyone except one participantsin the survey performed.
6.2.7 Binary approach, either you do it, or you don’t
The five activities, security requirements, Evaluation of COTS, Risk assess-ment, Attack surface analysis and Static Code analyzer are all positively
60
correlated to each other, see table 6.2. This means that when a participantperformed one of these activities, it was a high chance that he also performedthe others. Penetration testing and threat modelling is also positively cor-related to these activities as mentioned above. This suggest that our claimabout proactive software security being an on off switch, either a participantdoes many activities or they perform very few.
Table 6.2: Strong correlationsSecurity Re-quirements
EvaluateCOTS
Risk Assess-ment
Attack Sur-face Analy-sis
Static CodeAnalyzer
Security Re-quirements
1 .663** .805** .537* .701**
EvaluateCOTS
.663** 1 .645** .645** .832**
Risk Assess-ment
.805** .645** 1 .764** .620**
Attack SurfaceAnalysis
.537* .645** .764** 1 .588*
Static CodeAnalyzer
.701** .832** .620** .588* 1
From these results we can conclude that with the exception of reactivepatching and release plans, the focus on software security is really binary.Either the companies did perform many activities as table ?? shows, andtable B.1 shows that even more actitivies are correlated, further supportingthis claim.
6.2.8 Answering RQ2
To summarize the answer to RQ2. RQ2 states:
How is software security done in practice in Norwegian industrycompared to the state of art research and theory
The software security done in practice is mostly binary, meaning that eithera company does many of the state of the art research and theory activitiesor they perform very few. The big exception to this is patching. Almost allthe participants of the survey did have a way to easily patch the system incase vulnerabilities where discovered. The practice also shows some activitiesthat almost none did, which was fuzz testing, having a list of banned APIsand dynamic analysis.
Those who perform these activities therefore mostly follow the same principlesand guidelines as the state of the art frameworks suggests. One company inthe interview also had a security review, but this type of activity is mentionedin many of the frameworks and is also a consolidation of many differentactivities. The big answer to RQ2 will hence be that those who does proactive
61
security in the development follow many of the state of the art activitiesand principles, while those who do not perform proactive security in thedevelopment only does so for reactive patching of the software.
6.3 Which phases is the focus on
This subsection helps us in answering research question R2.1, Does softwaresecurity have the same focus in every stage of an applications lifecycle
6.3.1 Delivery
When we look at the yes and no answers from the survey, we can easilysee that at least everyone has a focus in the last phase, the deploymentphase of the software. Almost everyone in the survey had a way to patchand fix the system in case any serious threats was found. With this we seethat everybody had a big focus in the deployment phase of an applicationslifecycle.
6.3.2 Training
If we move all the way to the beginning we have a look at the trainingquestions. In figure 6.5 we have plotted the four types of training on theY-axis and the ids of the respondents at the X-axis. Because of some technicaldifficulties the ID starts at 3. Again we see that some people have a lot oftraining while some only have one type and many have zero types. If wecount those who did not receive any training last year against, we get 11.That means that 11 out of 21 did not receive any training during the lastyear. It therefore looks like about half of the projects have focus on trainingduring the training phase of the lifecycle.
6.3.3 Requirements and design and planning
If we move to the requirements phase. about half of the participants did havesecurity requirements, and about one fourth did a security review of COTS.Again about half of the projects did have some focus here. The same resultfor the design and planning phase. Approximately half of the participantsdid have a focus on security during this phase.
62
Figure 6.5: How many types of training each respondent had
6.3.4 Implementaion
Since there was many results in this phase, it is difficult to give concludeexactly how much focus the participants had on security in this phase. Aboutone fourth of all the participants did use a static code analyzer, while abouthalf did have a development handbook. None of the participants did haveany banned APIs. Almost all participants did perform pair programmingand they where almost divided 50/50 on how often they performed it. Abouthalf of the participants also did perform peer review. Because of these mixedresults it is hard to say much about the focus, but two of the activities havethe same result where about half of the participants performed it and halfdid not. Since we still had some results which where half, we can concludethat about half of the participants had a focus on security during this phase.
6.3.5 Verification and testing
In this last phase there was also a split with about half the participants, atleast on performing penetrations tests and security tests. There was however
63
very few who performed fuzz testing or dynamic analysis of the software.
6.3.6 Summary of focus during the phases
The focus on software security was the same in all but the last phase, abouthalf of the participants performed the most popular activities, while only aminority performed some of the less common ones. The big surprise here isthat almost all the participants did have a huge focus in the delivery phase,meaning that almost all of them was prepared to react in case there wheresome vulnerabilities discovered.
Research Questions RQ2.1 states:
Does software security have the same focus in every stage ofan applications lifecycle
To answer RQ2.1; about half of the participants in the survey did have afocus in all the phases, while all the participants had a focus in the last phaseof delivery phase. This suggests that only half of the participants have aproactive approach, while all have a reactive approach to software security.
64
Chapter 7
Conclusion
We will have the conclusion for the rearch questions as well as pointing outthings that needs further research before an answer is found.
7.1 Research Questions
We have answered the following research questions.
• RQ1: What are the theories and state of the art research available forsoftware security?
• RQ2: How is software security done in practice in Norwegian industrycompared to the state of art research and theory
– RQ2.2: Does software security have the same focus in every stageof an applications lifecycle
7.1.1 Research question 1
To answer the RQ1 we performed a small literature study. In this studywe focused on software security and more specifically on frameworks andguidelines which could be used when developing software. After performingthe literature study, we concluded with four big frameworks for softwaresecurity. We had found these referenced in many other papers and websites,and all these frameworks claimed to be much used in the industry. The fourbig frameworks where:
• Microsofts SDL
• OWASP CLASP
65
• Seven Touchpoints from Cigital/McGraw
• BSIMM
The thing these frameworks have in common, is that they work to enhancethe security of software being developed by modifying or changing the exist-ing software development process. Microsofts SDL presents entire securitydevelopment lifecycles for traditional waterfall projects, agile-projects andline-of-business projects. The four others have certain activities which fitsat different stages but it is up to the implementer of their framework totailor them to their development process. In chapter 2 we go through theseframeworks in detail and compare them to each other and have a discussionabout them.
The answer to RQ1 was therefore these big frameworks which we discov-ered from a small literature study about software security with a focus onframeworks that where widely used both in the industry as well as academia.
7.1.2 Research Question 2
To answer research questions RQ2 and it’s subquestion RQ2.1 we did aweb-based survey combined with some interviews. The study was givento 4 different companies in Norway and we got 19 complete answers and 3incomplete, meaning that some quit the survey midway through. The researchdone to answer RQ1 was used to construct the survey and ask questionsregarding software security and which activities the different companiesperformed.
From the survey results we got answers to both RQ2 and it’s subquestion.The results from the survey suggested that it was a binary approach tosoftware security, either a project did many of the activities from the fourbig frameworks or they performed almost none. We would also like to pointout that it was only 4 companies in the survey and the results could havebeen different with more companies answering the survey. RQ2.1 was alsoanswered with the results from the survey, one interesting result here is thatall the companies have a focus on security with patching and fixing problemsafter release. Beside this focus at the release it was the same as with the restof the survey, those which had a focus had a focus in all the phases of thedevelopment process while those who did not naturally did not have focus inany of the other phases.
The interviews was mostly used to check for things the survey did not coverand we had one interesting finding there. We found that one company didhave software security group which often did what they called a securityreview. They would come in on a project the company was having and
66
perform a security review, which included many of the activities the big fourframeworks mentions as well as BSIMM focus on just having a dedicatedsoftware security group.
7.2 Summary
In summary, we found the four big frameworks as an answer to RQ1. We usedthese to construct a web-based survey which we got 4 different companies toparticipate in. This was done in order to answer RQ2 and we did performsome interviews to check for things the survey did not cover. The answerto RQ2 was in short, that either a project in practice have a good focuson software security and performs many of the activities or they performsalmost none. The big exception here is patching of the system which all thecompanies did.
7.3 Further work
The survey we did was quite short and did not contain too many participantsnor did we know if we had equal representation from each of the participantssince the survey was completely anonymous. A suggestion is therefore todo a similar survey with more participants and see if the results are thesame. The survey could also go to greater length to ensure that it got anequal representation from the participants in that survey, meaning thatapproximately the same number of participants from different companies.
We got several interesting results from the survey which we currently donot possess an answer for. One of these things is that very few performedthreat modeling, while many for example did perform an analysis of theattack surface. Why we got these results we currently do not have any goodanswer to, and further research into this area could be interesting. Anotherinteresting result was the small focus on fuzz testing. This is also an areawhich are in need of further research to understand why so few did performfuzz testing on their system.
67
Bibliography
[Bec99] Kent Beck. Extreme Programming Explained: Embrace Change.The XP Series. Addison-Wesley, 1999.
[Bog96] K Bogen. The effect of questionnaire length on response rates:a review of the literature. Proceedings of the Section on SurveyResearch, pages 1020–1025, 1996.
[CA11] Brian Chess and Brad Arkin. Software Security in Practice.IEEE Security Privacy Magazine, 9(2):89–92, 2011.
[Cap09] Pete Cape. Questionnaire Length , Fatigue Effects and ResponseQuality Revisited. Survey Sampling International, (August),2009.
[CCL01] S D Crawford, M P Couper, and M J Lamias. Web Surveys: Per-ceptions of Burden. Social Science Computer Review, 19(2):146–162, 2001.
[CKMR03] Bill Cheswick, Paul Kocher, Gary McGraw, and Avi Rubin.Bacon Ice Cream: The Best Mix of Proactive and ReactiveSecurity?, 2003.
[DSB+09] Bart De Win, Riccardo Scandariato, Koen Buyens, JohanGregoire, and Wouter Joosen. On the secure software devel-opment process: CLASP, SDL and Touchpoints compared. In-formation and Software Technology, 51(7):1152–1171, July 2009.
[Fag76] ME Fagan. Design and code inspections to reduce errors inprogram development. IBM Systems Journal, 15(3):258–287,1976.
[Fag86] ME Fagan. Advances in Software Inspections. IEEE transactionson software engineering, 1986.
[Gui08] Owasp Testing Guide. OWASP Testing Guide v3.0. Search, (Cc),2008.
68
[HC02] Anthony Hall and Roderick Chapman. Correctness by construc-tion: developing a commercial secure system. IEEE Software,19(1):18–25, 2002.
[HL06] Michael Howard and Steve Lipner. The Security DevelopmentLifecycle, volume 34. Microsoft Press, 2006.
[HMA04] P Hope, G McGraw, and A I Anton. Misuse and abuse cases:getting past the positive. IEEE Security Privacy Magazine,2(3):90–92, 2004.
[IAT07] IATAC. SOAR, Software Security Assurance. Technical report,2007.
[KE02] DARRELL M KIENZLE and MATTHEW C ELDER. FinalTechnical Report : Security Patterns for Web Application De-velopment Security Patterns for Web Application Development.Defense, 2002.
[Lig06] Comprehensive Lightweight. Comprehensive, Lightweight Appli-cation Security Process (CLASP), 2006.
[McG06] Gary McGraw. Software security: building security in, 2006.
[McG09] Gary McGraw. OWASP Podcast, January 15, 2009.
[MCM09] G. McGraw, B. Chess, and S. Migues. Building Security InMaturity Model, 2009.
[MHN04] Jonathan D Moffett, Charles B Haley, and Bashar Nuseibeh.Core Security Requirements Artefacts. Practice, 23, 2004.
[Mic10] Microsoft. Simplified Implementation of the Microsoft SDL,2010.
[Mic11] Microsoft. Security Development Lifecycle SDL Process GuidanceVersion 5.1. 2011.
[OP07] An Oracle and White Paper. Oracle Software Security Assurance.Security, (January), 2007.
[Rin09] Kristen Ringdal. Enhet og mangfold: samfunnsvitenskapeligforskning og kvantitativ metode. Fagbokforlaget, 2009.
[SO00] G Sindre and A L Opdahl. Eliciting security requirements bymisuse cases. In Technology of ObjectOriented Languages andSystems 2000 TOOLSPacific 2000 Proceedings 37th InternationalConference on, volume 10, pages 120–131. IEEE Press, 2000.
69
[SSSB08] SG Shiva, TR Stoian, RR Satharla, and EA Ba. Towards anaugmented secure software development lifecycle. computersecu-rityconference.com, 2008.
[Tas02] G Tassey. The Economic Impacts of Inadequate Infrastructurefor Software Testing, 2002.
[YHSJ06] Koen Yskout, Thomas Heyman, Riccardo Scandariato, andWouter Joosen. A system of security patterns Katholieke Uni-versiteit Leuven Department of Computer Science A system ofsecurity patterns. Group, (December), 2006.
[YWM08] Nobukazu Yoshioka, Hironori Washizaki, and KatsuhisaMaruyama. A survey on security patterns. Progress In In-formatics, 5(5):35, 2008.
70
Appendix A
Survey Results
This appendix shows the results from the survey in one big table.
Table A.1: Results from the survey
Question and answers Count PercentageTotal records in survey: 22Percentage of total: 100.00%
In which industry does your project belong?Answer Count PercentageAgriculture, forestry and fisheries 0 0.00%Mining and quarrying 0 0.00%Manufacturing 1 4.55%Electricity, water and refuse disposal 0 0.00%Construction 1 4.55%Retail, repair of motor vehicles 0 0.00%Transport, storage and warehousing 3 13.64%Hotel and restaurant 0 0.00%Information and communications 6 27.27%Financial services and insurance 2 9.09%Technical activities, real estate 0 0.00%Business activities 1 4.55%Public administration, defence and social insurance 4 18.18%Education 1 4.55%Health and social work 2 9.09%Personal services 0 0.00%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 0 0.00%
Is this project developing a web application?Answer Count PercentageYes 21 95.45%No 1 4.55%Don’t know 0 0.00%No answer 0 0.00%Not completed or Not displayed 0 0.00%
What is the current size of the project, in terms of people involved?Answer Count Percentage1-5 8 36.36%5-10 4 18.18%10-20 5 22.73%
i
Question and answers Count Percentage20-40 3 13.64%40-60 1 4.55%60-80 1 4.55%80-100 0 0.00%100-150 0 0.00%150-200 0 0.00%200-250 0 0.00%250-300 0 0.00%300+ 0 0.00%No answer 0 0.00%Not completed or Not displayed 0 0.00%
What is the estimated timespan for the project?Answer Count PercentageLess than one month 1 4.55%2 months 2 9.09%4 months 0 0.00%6 months 3 13.64%8 months 0 0.00%10 months 1 4.55%1 year 4 18.18%1 and a half year 1 4.55%2 years 1 4.55%2 and a half year 0 0.00%3 years 3 13.64%4 years 0 0.00%5 years 2 9.09%6 years 0 0.00%7 years 0 0.00%8 years 0 0.00%9 years 1 4.55%10 years 0 0.00%More than 10 years 3 13.64%No answer 0 0.00%Not completed or Not displayed 0 0.00%
Who is responsible for the security of the software being produced?Answer Count PercentageCustomer security officer 2 9.09%Customer security group 2 9.09%Company security officer 4 18.18%Company security group 2 9.09%Security group from both customer and company 4 18.18%External security firm 0 0.00%Other 8 36.36%No answer 0 0.00%Not completed or Not displayed 0 0.00%
Is this person/groupAnswer Count PercentageExternal to the project 5 22.73%Internal to the project 5 22.73%Both internal and external to the project 3 13.64%Other 0 0.00%No answer 2 9.09%Not completed or Not displayed 7 31.82%
Which type of methology do you use for the development in thisproject?Answer Count PercentageTraditional Waterfall 2 9.09%Rational Unified Process RUP 1 4.55%Scrum 19 86.36%Lean Software Development 6, 7.27%
ii
Question and answers Count PercentageExtreme Programming 4 18.18
Other,5,22.73%
What is your role in this project?Answer Count PercentageChief Security Officer/ Chief Information Security Officer 0 0.00%Privacy Officer 0 0.00%Security Architect 1 4.55%Project Manager 5 22.73%Developer 16 72.73%Tester 0 0.00%Other 4 18.18%
How long have you been working on this project?Answer Count PercentageLess than 1 year 12 54.55%1-2 years 6 27.27%2-3 years 3 13.64%3-4 years 0 0.00%4-5 years 1 4.55%5-6 years 0 0.00%6-7 years 0 0.00%7-8 years 0 0.00%8-9 years 0 0.00%9-10 years 0 0.00%More than 10 years 0 0.00%No answer 0 0.00%Not completed or Not displayed 0 0.00%
How long have you been working in the software industry?Answer Count PercentageLess than 2 years 5 22.73%2-4 years 3 13.64%4-6 years 4 18.18%6-8 years 3 13.64%8-10 years 1 4.55%10-12 years 3 13.64%12-14 years 1 4.55%14-16 years 0 0.00%16-18 years 1 4.55%18-20 years 0 0.00%20-22 years 1 4.55%22-24 years 0 0.00%24-26 years 0 0.00%26-28 years 0 0.00%28-30 years 0 0.00%More than 30 years 0 0.00%No answer 0 0.00%Not completed or Not displayed 0 0.00%
Which types of Esecurity training have you received during the lastyear?Answer Count PercentageRead one or more books about security 7 31.82%Been to one or more conferences 6 27.27%Attended one or more security training courses 6 27.27%Read security news, blogs or listened to podcasts 8 36.36%Other 4 18.18%
Which security certifications do you have?Answer Count PercentageBachelor degree in information security 0 0.00%Master degree in information security 4 18.18%PHD degree in information security 0 0.00%CISSP, Certified Information Systems Security Professional 0 0.00%
iii
Question and answers Count PercentageCISM, Certified Information Security Manager 0 0.00%CompTIA Security+ 2 9.09%GIAC Global Information Assurance Certification 0 0.00%CEH Certified Ethical Hacker 0 0.00%Other 2 9.09%
How interested are you in security? [professionally]Answer Count Percentage1 1 4.55%2 1 4.55%3 0 0.00%4 1 4.55%5 5 22.73%6 2 9.09%7 4 18.18%8 3 13.64%9 2 9.09%10 2 9.09%No answer 0 0.00%Not completed or Not displayed 1 4.55%
How interested are you in security? [personally]Answer Count Percentage1 1 4.55%2 1 4.55%3 2 9.09%4 1 4.55%5 4 18.18%6 1 4.55%7 4 18.18%8 4 18.18%9 1 4.55%10 2 9.09%No answer 0 0.00%Not completed or Not displayed 1 4.55%
Do you have specific security requirements?Answer Count PercentageYes 9 40.91%No 12 54.55%Don’t know 0 0.00%Other 0 0.00%No answer 0 0.00%Not completed or Not displayed 1 4.55%
Do you review the security for OTS components?Answer Count PercentageYes 4 18.18%No 13 59.09%Don’t know 4 18.18%No answer 0 0.00%Not completed or Not displayed 1 4.55%
How to you review the OTS?Answer Count PercentageCheck with community 4 18.18%Review Documentation/Source code 3 13.64%Check the OTS in NVD or other vulnerability databases 4 18.18%Trusting the vendors testing of the component 0 0.00%Internal testing of the component 4 18.18%External testing of the component 0 0.00%Other 0 0.00%
Is this project certified with for example Common Criteria?No answer 19 86.36%
iv
Question and answers Count PercentageNot completed or Not displayed 1 4.55%
Do you perform a risk assessment to determine the biggest risks forthe software?Answer Count PercentageYes 9 40.91%No 10 45.45%Don’t know 1 4.55%No answer 0 0.00%Not completed or Not displayed 2 9.09%
Do you perform an analysis of the attack surface for the architectureof the software?Answer Count PercentageYes 8 36.36%No 9 40.91%Don’t know 3 13.64%No answer 0 0.00%Not completed or Not displayed 2 9.09%
Do you perform threat modelling for the architecture of the software?Answer Count PercentageYes 3 13.64%No 15 68.18%Don’t know 2 9.09%No answer 0 0.00%Not completed or Not displayed 2 9.09%
Who is doing the threat modeling and attack surface analysis?Answer Count PercentageExternal entity 0 0.00%Internal entity 1 4.55%Both external and internal entites 0 0.00%Other 1 4.55%No answer 2 9.09%Not completed or Not displayed 18 81.82%
Do you use any of these frameworks/methods when doing threat mod-eling?Answer Count PercentageAbuse cases or misuse cases 0 0.00%Data Flow Diagrams 0 0.00%Architecture diagram with trust boundaries 1 4.55%Attack Trees 0 0.00%STRIDE 1 4.55%DREAD 1 4.55%CVSS 0 0.00%OCTAVE 0 0.00%Other 1 4.55%
Do you use security patterns?Answer Count PercentageYes 7 31.82%No 12 54.55%Don’t know 1 4.55%No answer 0 0.00%Not completed or Not displayed 2 9.09%
When developing do you use a static code analyser that scans forpotential mistakes?Answer Count PercentageYes 5 22.73%No 14 63.64%Don’t know 0 0.00%Other 0 0.00%No answer 0 0.00%
v
Question and answers Count PercentageNot completed or Not displayed 3 13.64%
Which static code analyser do you use?Answer 5 22.73%No answer 0 0.00%Not completed or Not displayed 17 77.27%
Does the project have coding guidelines or developers handbook andis security mentioned there?Answer Count PercentageYes 9 40.91%No 9 40.91%Don’t know 0 0.00%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Does the project have a list of banned or dangerous APIs?Answer Count PercentageYes 0 0.00%No 19 86.36%Don’t know 0 0.00%Other 0 0.00%No answer 0 0.00%Not completed or Not displayed 3 13.64%
When coding do you often use pair programming?Answer Count PercentageNever 1 4.55%Sometimes 9 40.91%Only for hard features 0 0.00%Now and then 9 40.91%Always 0 0.00%Don’t know 0 0.00%Other 0 0.00%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Do you peer review all or parts of the source code before checking in?Answer Count PercentageYes 9 40.91%No 8 36.36%Don’t know 1 4.55%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Will the system undergo a penetration test before release?Answer Count PercentageYes 9 40.91%No 8 36.36%Don’t know 2 9.09%No answer 0 0.00%Not completed or Not displayed 3 13.64%
How is the penetration test performed? Check all that applyAnswer Count PercentageAn external entity is performing the pentest 6 27.27%The pentest is performed white box 4 18.18%The pentest is performed black box 4 18.18%The pentest is performed in the operational environment of the appli-cation
4 18.18%
There is a source code analysis as part of the pentest 4 18.18%Other 2 9.09%
Field summary for Dynamic analysisAnswer Count Percentage
vi
Question and answers Count PercentageYes 2 9.09%No 15 68.18%Don’t know 1 4.55%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Do you perform any fuzz testing of the system?Answer Count PercentageYes 3 13.64%No 14 63.64%Don’t know 2 9.09%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Check all that apply for fuzz testing in the projectAnswer Count PercentageEvery input field is fuzz tested 0 0.00%The fuzz test use well known malicious input 2 9.09%The fuzz test use random input 2 9.09%Other 0 0.00%
Do you have internal security tests which tests security features?Answer Count PercentageYes 7 31.82%No 11 50.00%Don’t know 0 0.00%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Is there a set of security requirements for the configura-tion/deployment?Answer Count PercentageYes 9 40.91%No 9 40.91%Don’t know 0 0.00%Other 1 4.55%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Is there a plan for handling bugs and problems after the release of thesoftware?Answer Count PercentageYes 15 68.18%No 2 9.09%Don’t know 2 9.09%Other 0 0.00%No answer 0 0.00%Not completed or Not displayed 3 13.64%
Is there a way to easily patch the system in case serious defects arefound?Answer Count PercentageYes 17 77.27%No 1 4.55%Don’t know 1 4.55%Other 0 0.00%No answer 0 0.00%Not completed or Not displayed 3 13.64%
vii
Appendix B
Correlations table
This is the big table containing all the correlations between different yes andno questions from the survey. See subsection 6.2.3 for information on how tointerpret these results.
viii
Table B.1: Correlations of Yes and No answers
Security Require-ments
Evaluate COTS Risk Assessment Threat Modelling
Security Requirements 1 .663** .805** .632**Evaluate COTS .663** 1 .645** .829**Risk Assessment .805** .645** 1 .500*Threat Modelling .632** .829** .500* 1Attack Surface Analy-sis
.537* .645** .764** .491*
Security Patterns .675** .447 .471* .378Static Code Analyser .701** .832** .620** .835**Development Guide .570* .535* .889** .378Peer Review .450 .408 .378 .419Dynamic Analysis .112 .294 .048 .429Penetration Testing .549* .564* .500* .535*Fuzz Testing -.019 .167 -.101 -.182Security Tests .433 .431 .290 .713**Requirements Operat-ing Environment
.342 .134 .292 .429
Patching .193 .161 .243 .116Release Plan .306 .175 .387 .182
Attack SurfaceAnalysis
Security Patterns Static Code Anal-yser
DevelopmentGuide
Security Requirements .537* .675** .701** .570*Evaluate COTS .645** .447 .832** .535*Risk Assessment .764** .471* .620** .889**Threat Modelling .491* .378 .835** .378Attack Surface Analy-sis
1 .167 .588* .630**
Security Patterns .167 1 .287 .570*Static Code Analyser .588* .287 1 .535*Development Guide .630** .570* .535* 1Peer Review .327 .450 .523* .417Dynamic Analysis .048 .112 .310 .022Penetration Testing .464 .549* .523* .500*Fuzz Testing -.101 -.019 -.214 -.127Security Tests .221 .532* .523* .290Requirements Operat-ing Environment
-.016 .570* .267 .333
Patching .236 .171 .150 .265Release Plan .378 .270 .236 .429
ix
Table B.2: Correlations of Yes and No answers continued
Peer Review Dynamic Analysis Penetration Test-ing
Fuzz Testing
Security Requirements .450 .112 .549* -.019Evaluate COTS .408 .294 .564* .167Risk Assessment .378 .048 .500* -.101Threat Modelling .419 .429 .535* -.182Attack Surface Analy-sis
.327 .048 .464 -.101
Security Patterns .450 .112 .549* -.019Static Code Analyser .523* .310 .523* -.214Development Guide .417 .022 .500* -.127Peer Review 1 .000 .875** .000Dynamic Analysis .000 1 .026 .310Penetration Testing .875** .026 1 -.134Fuzz Testing .000 .310 -.134 1Security Tests .775** .153 .882** -.324Requirements Operat-ing Environment
.764** .022 .630** -.127
Patching .258 .098 .a .124Release Plan .419 .154 .378 .196
Security Tests RequirementsOperating Environ-ment
Patching Release Plan
Security Requirements .433 .342 .193 .306Evaluate COTS .431 .134 .161 .175Risk Assessment .290 .292 .243 .387Threat Modelling .713** .429 .116 .182Attack Surface Analy-sis
.221 -.016 .236 .378
Security Patterns .532* .570* .171 .270Static Code Analyser .523* .267 .150 .236Development Guide .290 .333 .265 .429Peer Review .775** .764** .258 .419Dynamic Analysis .153 .022 .098 .154Penetration Testing .882** .630** .a .378Fuzz Testing -.324 -.127 .124 .196Security Tests 1 .783** .185 .293Requirements Operat-ing Environmen
.783** 1 .236 .378
Patching .185 .236 1 .aRelease Plan .293 .378 .a 1
**. Correlation is significant at the 0.01 level (2-tailed).*. Correlation is significant at the 0.05 level (2-tailed).
a. Cannot be computed because at least one of the variables is constant.
x
Appendix C
Interview Results
Below is a walkthrough of the results from the interview breaken into thesame phases as the survey is.
C.1 Project information
1. One of the projects interviewed was rather large, with over 50 persons withtechnical tasks with 25 of them being developers. It is now in maintenancebut there is still some new features being developed. This project is in thefinance and banking industry. The customer of this project have their ownsecurity group which takes responsibility for the security of the softwarebeing developed.
2. The next project was smaller, but still quite large, but we did not getany real number. The project was divided in two network zones, were allthe sensitive data being in the sensitive zone with no Internet access andall the other non-sensitive data being accessible from the Internet. For theweb-application to get access to the sensitive data it has to pass through alayer of security. Also in this project the customer has their own securitygroup which takes much responsibility for the security of the software beingdeveloped.
3. The third interview was a project with 7 people involved in the project.The project was much smaller, but had a big focus on code quality. Thisproject was in the finance industry. This project did not have anyone formallyresponsible but the interviewee was quite competent in security and thereforetook much of this responsibility.
4. This is also a small project with 4-6 developers. There are some peoplewhich only are there sometimes, and therefore it is difficult to measure the
xi
amount of people involved. The transportation industry is what the projectbelong to. The project did not have any formally security responsible butwas a relationship based on trust between the customer and consultants. Butin the end it was the customers responsibility is any major security breachwas going to happen.
5. The last project leader was leader for two projects, one with 6 developersand one with 3. The project was just one, but with two components, oneweb-application and one internally application. The project belonged to thepublic sector in Norway. In this project the developers themselves had theresponsibility for the security.
The responses from the interview do not differ much with regards to theprojects and whom is responsible for the security. On the larger projectswith there often is a security group or chief, while on smaller projects this isoften up to the developers themselves. This is the same trend as we saw inthe survey.
C.2 Training and education
None of the projects interviewed had any security requirements to certifi-cations or training regarding the people involved in the project. None ofthe projects involved did have any certification for the software itself, likecommon criteria, this is also aligned with the results from the survey.
C.3 Requirements
In the survey, 9 out of 12 answered that they had specific security requirementsin their project which is quite a lot, the trend in the survey was that all thebig projects had security requirements, while only some of the smaller onesin the survey had.
1. The first interview on the large project had many specific security require-ments. The interviewee did not know about them all, since he did not workwith security alone, but more with the project as a whole. The usage ofCOTS where handled by the customers security group which had a thoroughlook on COTS and other libraries used in the project.
2. The second project did not mention any security requirements during theinterview, but they could very well be there. However they where responsiblefor the inclusion of frameworks and other COTS. They did have a look intodifferent vulnerability databases when choosing COTS, and also used the
xii
history and maturity of the COTS and frameworks to validate it for use inthe project.
3. This project did not have any specific security requirements. This is justlike the other small projects in the survey, which did not have any securityrequirements either. This project had however a requirement that anyinternet facing application should undergo a penetration test before release.This was however quite relaxed in practise and only done approximatelyonce a year. The project did not contain any formal review of the COTSand frameworks they used, but they only used well known and well usedframeworks.
4. The fourth project is much like the third, there is no formal securityrequirements but the customer have a good relationsship with the consultantcompany delivering the software and trusts them to make it secure. Mostof the COTS used in this project are well known frameworks, but theyconsidered taking in a smaller project, but did a small code review andthe code did not match their standards. So when they do not trust thevendor or project which makes the COTS or frameworks they did some moreinvestigation of the code before rejecting it.
5. The last project was also quite small, but in contrast to three and four,this project had specific security requirements. This is also in line with thefindings from the survey were some of the small project also had securityrequirements and others did not. When it comes to review of COTS andframeworks, this project had a prestudy before the project started. In thisprestudy all the frameworks which were to be used was reviewed and altoughthere were no formal analysis on security, it was a consideration.
The security requirements, review of COTS and frameworks matches thesurvey quite well. All the large projects interviewed did all have specificsecurity requirements, while only some of the smaller ones did.
C.4 Design and planning
In the survey, about half of the participants answered that they did a riskassessment and performed an analysis of the attack surface. However veryfew said they did a threat model of the architecture of the application. Onthe other hand a little under half again said that they did used securitypatterns in their development.
1. The first and biggest project did perform regular risk analysis. Securityand availability was one of the biggest priorities of this project. This projectalso did make threat models of the architecture and the attack surface. The
xiii
interviewee did not know if the project used security patterns but theyguessed they did.
2. The second project did also perform risk analysis. They did not haveany threat models but they performed a security review by both internaland external entities. The external performed a penetration test, and theinternal review did go through the source code both with tools and manually.The internal security review also trained the developer in the project in theuse of secure coding practices. This project did also perform an analysis ofthe threats during the risk analysis, this is something the survey did notask about but we learned about in the interview. They did not use anysecurity patterns, but they used misuse cases to help identify the threat tothe application.
3. This project did have a risk analysis, but did not perform any formalthreat modeling. However since the project was quite small, there was nota great need for a formal threat model of the architecture. They did notformally used security patterns, but they used a couple of best practices,such as security in depth, security gateway which all sensitive informationhad to pass through.
4. The fourth project did not perform any formal risk or threat analysis.They had performed a analysis of the threats in the past and concluded thatthey did not need too much focus on formal activities regarding security.
5. This last project did perform some threat modeling, but not much. Theyalso had a simple risk analysis, but nothing big. This was mostly becausethey had rewritten much of the architecture and not changed much on theattack surface, and therefore they did not have much need to perform threatmodeling or risk analysis. We did however not find out if they used securitypatterns or not.
As we can see many of the projects interviewed not all of them did perform arisk analysis, reduction of the attack surface or threat modeling. This is thesame results as from the survey. We see that the bigger project have more ofa focus on formal activities while the smaller projects do not necessarily haveany formal activities with regards to risk analysis, attack surface reduction orthreat modeling. The results are almost the same for the security patterns,the bigger project used them explicitly while the smaller ones often just usedbest practices they knew about or did not use security pattern at all.
C.5 Implementation
In the survey, about half of the participants used static code analyzers andhad a developer handbook or coding guidelines which mentioned security.
xiv
Interestingly none of the project did have any banned APIs. All the projectsexcept one did utilize pair-programming, but the rate at which they utilizedit differed. About half of the project also performed peer-review of codebefore it being checked in into the source code control system.
1. In the first project, the interviewee did not know if there was any staticcode analyzers being used, however they did have coding guidelines whichmentions security. As with the survey they did not have any banned APIs.The interviewee did not know if there was any pair-programming or peer-review of code.
2. The second project did not have much information about the implementa-tion details. They did not have a developers handbook since they tried tokeep the documentation at a minimum, they did however have courses andother activities to enhance the knowledge of the developers. We don’t knowif they used any static code analysis tools or not.
3. The third project did exercise pair-programming and peer-review on some,but not all the code written. The interviewee did use a static code analysistool, but the other developers on the project did not use it. The project didnot have any developers handbook or anything like it.
4. The fourth project exercise both pair-programming and peer-review. Ifsomething is not pair-programmed then they execute a peer-review to ensurethat at least two people have reviewed the code before it is checked in. Theproject does use a simple static code analyzer for Javascript code, but theydo not have any other static analyzers. However the interviewee specifiesthat the project have a big focus on code quality, they do not take anyshortcuts. The project does not have any developers handbook, but theyhave guidelines for how the code should be written, but not anything specificabout security.
5. The last project does execute pair-programming to a large extent, butdoes not have any peer-review. They do not have any developers handbook.The interviewee did not know if there was any static code analysis toolsbeing used, but he did not think so.
As we can see, many of the project does pair-programming and some ofthem does peer-review. This is just like the results from the survey. Onlythe biggest project did have some coding guidelines which mentions security.The usage of static code analyzers was at a minimum, but some projectsdid use them. This is a bit puzzling since it is a very cost efficient way ofdiscovering bugs in the software being produced.
xv
C.6 Testing and verification
From the survey, about half of the projects asked did perform a penetrationtest before the software was released. Only two of the participants didperform a dynamic analysis of the software. There were three participantswho performed fuzz-testing of the software. 8 out of 19 did perform securitytests which tested security features of the software. Now let us comparethese results to the interviews.
1. The first project did have external company which performed botha penetration test, whitebox code analysis which produces a report andrecommendations of how to mitigate the security vulnerabilities discoveredby the external company. The interviewee did not know if there was anyfuzz-testing but he assumed it was. He did not know about any dynamicanalysis but since it was a large project it could very well be done.
2. The second project did have both internal and external security review.Penetration tests where done by external company. The penetration test wasmostly done black box but the company performing the test did have accessto resources on the inside. The security review was done by both interviewswith the developers as well as a walk trough with both the developers andthe people executing the security review. Fuzz-testing was not much used.It was not performed any dynamic analysis of the software though.
3. The third project did contain both penetration testing which involvedsome fuzz-testing. The penetration test was performed internally by thecustomers security group. We did not find out if they had any security testswhich tested security functionality.
4. This project did not perform any penetration tests. They did howeverperform a security review which walked through much of the code and didan analysis of the software. The project did not have any fuzz-testing norany dynamic analysis.
5. The last project have performed a penetration test once. They also hadsome external consultants which reviewed the software being produced. Inaddition to this they also had an internal security review of the software.They did not have any fuzz testing nor dynamic analysis.
As we can see from these results, many performs penetration testing butvery few does any fuzz testing. This is the same results as we got from thesurvey. However, many of the projects does have a security review, we willmention it more in details after we have gone through the last phase anddiscuss how this security review compares to the four big frameworks usedto create the survey.
xvi
C.7 Delivery
About half of the project in the survey did have specific security requirementsfor their operational environment. Almost all the project in the survey didhave a release plan and an easy way to patch the system in case therewere discovered some security vulnerabilities. Let us see how the interviewscompares to this.
1. The first project did have both requirements to the operational environ-ment as well as regular patching of both the software they develop as well asthe operational environment itself. The operational environment was hostedby the customer and their operations group.
2. The second project did also have specific security requirements for theiroperational environment. The environment was hosted by the customerand they already had separate network zones for trusted and untrustedtraffic. The consultant company developing the software was responsible forframeworks used in their operational environment, and they had routines topatch them if the need arises.
3. The third project did also have many security requirements for theoperational environment. Since this was a maintenance project many ofthese requirements was already in place. They had a plan for how to patchsecurity vulnerabilities in case they where discovered. However as part ofthat plan was a risk analysis to determine if the vulnerability was bad enoughthat a hot-fix was required.
4. We did not find out during the interview if there was any securityrequirements for the operational environment. The project did however haveroutines for patching the system in case of security vulnerabilities. This hadhappened in the past and they had rolled out a hot-fix quickly.
5. The fifth project had security requirements for the operational environment.They also developed on an old system with good routines for fixing problemthat could be discovered in production.
Almost all the project in the interviews did have specific security requirements,opposed to only about half of the projects in the survey. This can be since 4out of 5 many of the interview was done with the same consultant company,the survey might have reached a broader audience than the interviews. Allthe project in the interview did have routines to fix vulnerabilities that wherediscovered in the application and patch them. This is the same result as inthe survey.
xvii
Appendix D
Survey
This Appendix shows the survey with all it’s questions. The survey is dividedinto it’s questions groups which is roughly equivalent to a traditional waterfallprocess.
xviii
Survey about Software SecuritySurvey about the current state of Software Security in the norwegian industry
This is an anonymous survey taking a look at the current practice of software security in the industry. The survey is partof a specialization project in preparation for the master thesis at NTNU. The specialisation project is to compare state-of-the-art research into software security and with the actual practise used in the industry in Norway.
Software Security is the field of building security into the software itself, meaning that software should be able to resistand function normally even during an attack.
The survey is about the current project you are working on
Estimated time to complete: 15 Minutes
There are 38 questions in this survey
Project Information
Some practical information before the survey starts, again all this information is anonymous.
1 [industry]
In which industry does your project belong?
*
Please choose only one of the following:
Agriculture, forestry and fisheries
Mining and quarrying
Manufacturing
Electricity, water and refuse disposal
Construction
Retail, repair of motor vehicles
Transport, storage and warehousing
Hotel and restaurant
Information and communications
Financial services and insurance
Technical activities, real estate
Business activities
Public administration, defence and social insurance
Education
Health and social work
Personal services
1 of 18 12/5/11 19:13
Other
2 [webbased]Is this project developing a web application? *
Please choose only one of the following:
Yes
No
Don't know
3 [Size]What is the current size of the project, in terms of peopleinvolved? *
Please choose only one of the following:
1-5
5-10
10-20
20-40
40-60
60-80
80-100
100-150
150-200
200-250
250-300
300+
4 [TimeProject]What is the estimated timespan for the project? *
Please choose only one of the following:
Less than one month
2 months
4 months
6 months
8 months
10 months
1 year
2 of 18 12/5/11 19:13
1 and a half year
2 years
2 and a half year
3 years
4 years
5 years
6 years
7 years
8 years
9 years
10 years
More than 10 years
5 [SecurityOfficer]Who is responsible for the security of the softwarebeing produced? This can be either the customer or the companydelivering the software. *
Please choose only one of the following:
Customer security officer
Customer security group
Company security officer
Company security group
Security group from both customer and company
External security firm
Other
3 of 18 12/5/11 19:13
6 [SecurityOfficer3]Is this person/group
Only answer this question if the following conditions are met:° Answer was `A1`'Customer security officer' or 'Customer security group' or 'Company security officer' or'Company security group' or 'Security group from both customer and company' at question '5 [SecurityOfficer]'(Who is responsible for the security of the software being produced? This can be either the customer or thecompany delivering the software.)
Please choose only one of the following:
External to the project
Internal to the project
Both internal and external to the project
Other
7 [Methology]
Which type(s) of methology do you use for the development in thisproject?
You can choose multiple methologies, because some projects oftencombine methodologies for the implementation, for example RUP withscrum sprints.
*
Please choose all that apply:
Traditional Waterfall
Rational Unified Process RUP
Scrum
Lean Software Development
Extreme Programming
Other:
8 [position]
What is your role in this project?
Some of the more common roles are listed, but please add any other rolesyou have
*
Please choose all that apply:
4 of 18 12/5/11 19:13
Chief Security Officer/ Chief Information Security Officer
Privacy Officer
Security Architect
Project Manager
Developer
Tester
Other:
9 [projecttime]How long have you been working on this project? *
Please choose only one of the following:
Less than 1 year
1-2 years
2-3 years
3-4 years
4-5 years
5-6 years
6-7 years
7-8 years
8-9 years
9-10 years
More than 10 years
10 [Time in industry]How long have you been working in the softwareindustry? *
Please choose only one of the following:
Less than 2 years
2-4 years
4-6 years
6-8 years
8-10 years
10-12 years
12-14 years
14-16 years
5 of 18 12/5/11 19:13
16-18 years
18-20 years
20-22 years
22-24 years
24-26 years
26-28 years
28-30 years
More than 30 years
6 of 18 12/5/11 19:13
Training and education
This section covers security training.
11 [training last year]Which types of security training have you receivedduring the last year?
Please choose all that apply:
Read one or more books about security
Been to one or more conferences
Attended one or more security training courses
Read security news, blogs or listened to podcasts
Other:
12 [certification]Which security certifications do you have?
Please choose all that apply:
Bachelor degree in information security
Master degree in information security
PHD degree in information security
CISSP, Certified Information Systems Security Professional
CISM, Certified Information Security Manager
CompTIA Security+
GIAC Global Information Assurance Certification
CEH Certified Ethical Hacker
Other:
13 [Interest]How interested are you in security? *
Please choose the appropriate response for each item:
1 2 3 4 5 6 7 8 9 10professionallypersonally
7 of 18 12/5/11 19:13
Requirements
14 [SecurityRequirements]
Do you have specific security requirements?
Security requirements can for example be that all sensitive data shall bestored encrypted
*
Please choose only one of the following:
Yes
No
Don't know
Other
15 [COTS]
Do you review the security for OTS (Off-the-shelf) components?
Example of OTS: Frameworks, libraries, for example Spring, Hibernate,Linq
*
Please choose only one of the following:
Yes
No
Don't know
16 [COTS2]How to you review the OTS?
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '15 [COTS]' ( Do you review the security for OTS (Off-the-shelf)components? Example of OTS: Frameworks, libraries, for example Spring, Hibernate, Linq )
Please choose all that apply:
Check with community (forums, maillists, etc)
Review Documentation/Source code
Check the OTS in NVD or other vulnerability databases
Trusting the vendors testing of the component
Internal testing of the component
8 of 18 12/5/11 19:13
External testing of the component
Other:
17 [Certifications]Is this project certified with for example CommonCriteria? Write down certifications that may apply to this project.
Please write your answer here:
9 of 18 12/5/11 19:13
Design and planning
18 [Risk Assessment]Do you perform a risk assessment to determine thebiggest risks for the software? *
Please choose only one of the following:
Yes
No
Don't know
19 [AttackSurfaceArch]
Do you perform an analysis of the attack surface for the architecture of thesoftware?
The attack surface can for example be the number of entry points into theapplication
*
Please choose only one of the following:
Yes
No
Don't know
20 [ThreatModelArchitect]
Do you perform threat modelling for the architecture of the software?
A threat model is a model of all the different threats to the software. Ashort list of threat modeling:
Identify assets.Create an architecture overview.Decompose the application.Identify the threats.Document the threats.Rate the threats.
*
Please choose only one of the following:
Yes
10 of 18 12/5/11 19:13
No
Don't know
21 [Threatmodelingext]Who is doing the threat modeling and attacksurface analysis?
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '20 [ThreatModelArchitect]' ( Do you perform threat modelling for thearchitecture of the software? A threat model is a model of all the different threats to the software. A short list ofthreat modeling: Identify assets. Create an architecture overview. Decompose the application. Identify thethreats. Document the threats. Rate the threats. )
Please choose only one of the following:
External entity
Internal entity
Both external and internal entites
Other
22 [ThreatModelArchitec2]Do you use any of these frameworks/methodswhen doing threat modeling?
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '20 [ThreatModelArchitect]' ( Do you perform threat modelling for thearchitecture of the software? A threat model is a model of all the different threats to the software. A short list ofthreat modeling: Identify assets. Create an architecture overview. Decompose the application. Identify thethreats. Document the threats. Rate the threats. )
Please choose all that apply:
Abuse cases or misuse cases
Data Flow Diagrams (DFD) with trust boundaries
Architecture diagram with trust boundaries
Attack Trees
STRIDE
DREAD
CVSS
OCTAVE
Other:
23 [Patterns]
Do you use security patterns?
11 of 18 12/5/11 19:13
A security pattern is like a design pattern but intended to reach a goal inthe area of security
*
Please choose only one of the following:
Yes
No
Don't know
12 of 18 12/5/11 19:13
Implementation
Questions regaring security while you are performing the implementation
24 [CodeAnalyser]When developing, do you use a static code analyser thatscans for potential mistakes? *
Please choose only one of the following:
Yes
No
Don't know
Other
25 [codeanalyser2]Which static code analyser do you use?
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '24 [CodeAnalyser]' (When developing, do you use a static code analyserthat scans for potential mistakes?)
Please write your answer here:
26 [DevelopmentGuide]Does the project have coding guidelines ordevelopers handbook and is security mentioned there? *
Please choose only one of the following:
Yes
No
Don't know
Other
27 [BannedAPI]Does the project have a list of banned or dangerous APIs?
13 of 18 12/5/11 19:13
*
Please choose only one of the following:
Yes
No
Don't know
Other
28 [PairProgram]When coding, do you often use pair programming? *
Please choose only one of the following:
Never
Sometimes
Only for hard features
Now and then
Always
Don't know
Other
29 [Review]
Do you peer review all or parts of the source code before checking in?
Peer review means that in addition to the author, one or more people lookthrough the code
*
Please choose only one of the following:
Yes
No
Don't know
Other
14 of 18 12/5/11 19:13
Testing and Verification
30 [Pentest]Will the system undergo a penetration test before release? *
Please choose only one of the following:
Yes
No
Don't know
31 [PenTest2]How is the penetration test performed? Check all that apply
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '30 [Pentest]' (Will the system undergo a penetration test before release?)
Please choose all that apply:
An external entity is performing the pentest
The pentest is performed white box
The pentest is performed black box
The pentest is performed in the operational environment of the application
There is a source code analysis as part of the pentest
Other:
32 [Dynamic analysis]
Do you use any tools for run-time dynamic analysis?
This include tools that monitor application behavior for memorycorruption, user privilege issues, and other critical security problems. TheMicrosoft Security Development Lifecycle uses run-time tools likeAppVerifier.*
Please choose only one of the following:
Yes
No
Don't know
Other
33 [Fuzz1]
15 of 18 12/5/11 19:13
Do you perform any fuzz testing of the system?
Fuzz testing is a testing technique, that involves providing invalid,unexpected or random input to a software application. The application isthen monitored for crashes or misbehaviour
*
Please choose only one of the following:
Yes
No
Don't know
34 [Fuzz2]Check all that apply for fuzz testing in the project
Only answer this question if the following conditions are met:° Answer was `A1`'Yes' at question '33 [Fuzz1]' ( Do you perform any fuzz testing of the system? Fuzz testing isa testing technique, that involves providing invalid, unexpected or random input to a software application. Theapplication is then monitored for crashes or misbehaviour )
Please choose all that apply:
Every input field is fuzz tested
The fuzz test use well known malicious input
The fuzz test use random input
Other:
35 [Security tests]Do you have internal security tests which tests securityfeatures? *
Please choose only one of the following:
Yes
No
Don't know
Other
16 of 18 12/5/11 19:13
Release and maintainance
36 [Operating environmen]
Is there a set of security requirements for the configuration/deployment?
A simple example of a requirement: All traffic shall be over TLS/SSL.
*
Please choose only one of the following:
Yes
No
Don't know
Other
37 [Release plan]Is there a plan for handling bugs and problems after therelease of the software? *
Please choose only one of the following:
Yes
No
Don't know
Other
38 [Patching]Is there a way to easily patch the system in case seriousdefects are found? *
Please choose only one of the following:
Yes
No
Don't know
Other
17 of 18 12/5/11 19:13
Thanks for participating in this survey!
If you have any questions, please contact me at the email-adress [email protected]
01.01.1970 – 01:00
Submit your survey.Thank you for completing this survey.
18 of 18 12/5/11 19:13
