8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
1/56
Session #H8
, ,
11:30am
Muema Lombe, CRISC, CSSLP, CGEIT, CISA
IT Audit Manager
Endurance Services Ltd.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
2/56
Key Points
The many reasons projects fail
Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & samplefindin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
3/56
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
4/56
Project Management Methodologies
PMI PMBOK A Guide to the Project Management Body of Knowledge
(PMBOK Guide) is a book which presents a set of standardterminology and guidelines for project management.
It was first published by the Project Management Institute in
1987. PMI lobal standards rovide uidelines, rules and
characteristics for project management.
http://www.pmi.org/
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
5/56
Project Management Methodologies
Prince2 PRojects IN Controlled Environments (PRINCE) is a project
management method. It covers the management, control andorganisation of a project. "PRINCE2" refers to the secondmajor version of this method and is a registered trademark ofthe Office of Government Commerce (OGC), an independentoffice of HM Treasury of the United Kingdom.
http://www.prince-officialsite.com/home/home.asp
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
6/56
Project Management Methodologies
Waterfall The waterfall model is a sequential design process, often
used in software development processes, in which progressis seen as flowing steadily downwards (like a waterfall)through the phases of Conception, Initiation, Analysis,Design, Construction, Testing and Maintenance.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
7/56
Project Management Methodologies
The "waterfall model". Progress flows from the top to the bottom, like a waterfall.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
8/56
Key Points
The many reasons projects fail Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & samplefindin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
9/56
The Many Reasons Project Fail
Failed project example: Denver Airport Baggage System
Early Warning Signs of Project Failure
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
10/56
The Many Reasons Project Fail
Denver Airport Baggage System Case Study
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
11/56
The Many Reasons Project Fail
Denver Airport Baggage System Case Study
Originally billed as the most advanced system in the world, thebaggage handling system at the new Denver InternationalAirport was to become one of the most notorious examples ofproject failure. Originally planned to automate the handling ofbaggage through the entire airport, the system proved to be farmore complex than some had original believed. The problemsbuilding the system resulted in the newly complete airport
sitting idle for 16 months while engineers worked on getting the.
The delay added approximately $560M USD to the cost of theairport
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
12/56
The Many Reasons Project Fail
Denver Airport Baggage System Case Study
The Denver debacle is a tem late for failure that man other ro ects have followed.As with so many other failures, Denver suffered from:
1. The underestimation of complexity
2. A lack of lannin resultin in subse uent chan es in strate
3. Excessive schedule pressure
4. Lack of due diligence
5. Making firm commitments in the face of massive risks and uncertainty.
7. Communications breakdowns
8. People working in silos
9. Poor design
. a ure o per orm r s managemen
11. Failure to understand the implication change requests might have
12. Lack of management oversight
2011 Muema Lombe
Source: http://calleam.com/WTPF/wp-content/uploads/articles/DIABaggage.pdf
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
13/56
The Many Reasons Project Fail
2011 Muema Lombe
Source: EARLY WARNING SIGNS OF IT PROJECT FAILURE: THE
DOMINANT DOZEN by Leon A. Kappelman, Robert McKeeman, andLixuan Zhang
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
14/56
Key Points
The many reasons projects fail Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & samplefindin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
15/56
Project Audit Roles & Responsibilities
IT Audit Role on Pre/Post ImplementationReviews
IT Audit Level of Engagement
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
16/56
Project Audit Roles & Responsibilities
PMO Project Role to monitor and report project status, andto monitor and report project costs.
IT Internal Audit Project Role opine on the overall controlenvironment of the new project, system or application byevaluating, testing, and commenting on the effectiveness ofrisk management, control and governance processesfocusing on technology risks.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
17/56
Project Audit Roles & Responsibilities
Reviews To ensure new systems, or applications include all
consultative and proactive manner by evaluating,testing, and commenting on the effectiveness of risk
management, control, and governance processes.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
18/56
Project Audit Roles & Responsibilities
The level and depth of IT audit involvement in preand post implementation reviews is based on the
ro ect risk assessment,
project team's project management experience,
level of management involvement,
size and complexity of the initiative, and mpac on e organ za on e n a ve s e aye or
unsuccessfully implemented.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
19/56
Project Audit Roles & Responsibilities
Internal auditors' involvement in an organization's system conversion initiatives can range from minimal involvement (level1) to extensive audit efforts throughout every phase of the project (level 10).
Level 1: Audit risk assessment during the project initiation phase.
Level 23: Audit review of documentation and project deliverables.
Level 45: Attend project meetings, conduct some interviews, and produce verbal audit reports. Level 67: Increased audit efforts, conduct more interviews, and produce formal audit reports.
Level 89: Review all milestones, perform extensive audit tests, and produce formal and comprehensive audit reports.
.
Internal auditors should determine their level of involvement and approach during the project's initiation phase. The auditinvolvement decision should be based on the project risk assessment, as well as factors such as the project team's
project management experience, level of management involvement, size and complexity of the initiative, and impact.
needs to be defined during the audit project planning phase. Following step 1 of the generic eight-step audit processwill complete the definition of audit's level of involvement. Further adjustment of audit involvement may be requireddepending on the results of the projects efforts and auditors' assessment throughout the project's life cycle. Anotherimportant consideration to discuss with management and the project team is the internal auditors' roles andres onsibilities in attendin ro ect team meetin s throu hout the conversion audit.
Source: IIA.org
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
20/56
Project Audit Roles & Responsibilities
Reporting Once of the level of engagement has been decided, agree on the format,
.
DISTRIBUTION: Will reports be distributed to all project participants? Projectsponsor? Project steering committee?;
FREQUENCY: Will reports be prepared quarterly, monthly, or at the end of thepre pos mp emen a on rev ew
FORMAT: Agree on format - will the end result be a format audit report? If areport, will it be a detailed report? Executive summary? Will the end result be a
memo with observations rather than a report with issues?
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
21/56
Key Points
The many reasons projects fail Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & samplefindin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
22/56
Understanding Project Risk Factors
2011 Muema Lombe
Source: Project risk factors checklistwww.techrepublic.com
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
23/56
Understanding Project Risk Factors
2011 Muema Lombe
Source: Project risk factors checklistwww.techrepublic.com
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
24/56
Key Points
The many reasons projects fail Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & samplefindin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
25/56
Case Studies of Project Audits & Sample
Case #2: Regulatory Reporting Platform
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
26/56
Case Studies of Project Audits & Sample
customers to view, filter and report on data. Audit Involvement: IT audit participated in
wee y pro ec eam mee ngs as an o server,and prepared a quarterly memo summarizing
project status and observations. Findings:
Resource Management
Software Development Process
Vendor Management
Scope & Requirements Management
Business Participation
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
27/56
Case Studies of Project Audits & Sample
Findings:
Resource Management
Business analysts shortage of team members with the right skill sets to write
requirements. Project managers team members did not have sufficient experience running
large, complex projects.
Technical architect no one with end-to-end responsibility for overall technicaldesign
Priorities ongoing resource conflicts due to competing priorities and lack ofdedicated project resources.
Offshoring occurred during critical development time period and disruptedmanagements project focus.
Roles & responsibilities given the above issues, the role of team memberswas unclear and led to an over-reliance on mana ement b consensus durin
the requirements process.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
28/56
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
29/56
Case Studies of Project Audits & Sample
Findings:
Vendor Management
Vendor Selection insufficient screening of 3rd party software packages.
Vendor Oversight inadequate management of consultants assigned criticaldevelopment work.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
30/56
Case Studies of Project Audits & Sample
Findings:
Scope & Requirements Management
Project Scope the project scope was not realistic for the timeframe requested.
Requirements expertise analyst team members lacked adequate training inrequirements development and were not dedicated to the task.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
31/56
Case Studies of Project Audits & Sample
Findings:
Business Participation
Steering Committee the steering committee served more as a working group
than a senior management oversight committee. As a result, the key businessstakeholders were not included, making it difficult to insure they received earlynotice of emerging problems.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
32/56
Case Studies of Project Audits & Sample
regulatory reporting in 3 European countries. Audit Involvement: IT audit participated in the
pro ec y mee ng w e eam ea mon yand reviewing available project
documentation. IT audit prepared a monthly
observations.
Findings:
Project Management Design Documentation
Resources
2011 Muema Lombe
ppage
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
33/56
Case Studies of Project Audits & Sample
Findings:
Project Management
Lack of a Project Management Resource - Six months into an 18 month project,
a project manager has not been assigned to the project, as a result, projectmonitoring and tracking is not being performed.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
34/56
Case Studies of Project Audits & Sample
Findings:
Design Documentation
Inadequate Design Documentation - Detailed design documentation was not
prepared. A flowchart served as the sole design document. The flowchartcontained limited details with vague explanation of how to receive data fromsource systems and further process the data.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
35/56
Case Studies of Project Audits & Sample
Findings:
Resources
Inadequate Project Resources In addition to the lack of a dedicated project
manager, 2 resources are expected to document regulatory reportingrequirements for 3 countries, in addition to their daily responsibilities.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
36/56
Case Studies of Project Audits & Sample
Findings:
Slippage
Project slippage IT audit noted 2 weeks of slippage based on the original
project plan. The plan does not include any timing for contingency, as such thiscould impact the timely delivery of the project. Additionally, this slippage wasnot communicated to the Project Sponsor such that appropriate remedial actioncou e a en.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
37/56
Key Points
The many reasons projects fail Project audit roles & responsibilities
Understanding project risk factors
Case studies of project audits & sample
findin s
The key components of a project auditprogram
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
38/56
The Key Components of a Project Audit Program
Project Sponsorship
Project Organization
Quality Assurance Management
Procurement Management
Project Milestones
Scope Measurement
Project Approach Management
Schedule Management
3rd Party Management
Communication Management
Interdependency Management
Change Management
Cost Management
Personnel Resource Management
Risk Management
Transition Management
Project Conclusion Management
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
39/56
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
40/56
The Key Components of a Project Audit Program
Project Organization (Roles and Responsibilities)
This element refers to how the team is structured, reporting relationships, liaison roles. It isg y es ra e a user epar men s a e nc u e w n e pro ec eam n or er o
ensure that the system fully reflects user needs and to obtain greater user commitment to the
system. Some aspects of the project may be delegated to outside consultants andcontractors. It is essential in each case to specify the tasks and deliverables to be providedb each art , and the dates b which the must be com leted.
Audit should verify the roles and responsibilities have been defined and appropriate tasksand deliverables have been defined.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
41/56
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
42/56
The Key Components of a Project Audit Program
Scope Measurement
The element refers to ensuring the scope and objectives must be defined properly. Thesecan e groupe un er wo ma n ea ngs:
The project charters, which defines project goals and contains an outline proposal for the
project. This outlines describes the proposed system and the information it will use. Costs,benefits, preliminary schedule, and the impact the system will have on the organization and
.
The enterprise description is an important component, especially on larger projects withmany people involved. It lays out the users need and environment and relates them to thoseof the organization and external factors. Without an enterprise description there is a greater
chance that users real needs and the overall requirements of the organization will not besatisfied.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
43/56
The Key Components of a Project Audit Program
Project Approach Management
It is highly desirable for the project to be defined formally in a document that describes itsscope an e ro es o ose nvo ve . xper ence as s own a users w o are prepare oproceed on an informal basis have little understanding of what is involved in implementing
the system.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
44/56
The Key Components of a Project Audit Program
Schedule management
This element refers to the process of allocating activities (identified in the breakdown andgiven estimated times) to members of a project team against a calendar. Considerations forskill level ro ect team structure and size and schedulin and bud etin resources shouldbe included in the development methodology.
Audit should ensure the necessary resources are allocated to the project and.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
45/56
The Key Components of a Project Audit Program
Cost Management
This element relates to ensuring the initial pre-determined costs remain at or below projectedcos s. orma y on a arge sca e pro ec , a res o w e es a s e , a w e nc u ein the project charter, which will state if a project is going to exceed projected costs by a
certain dollar amount or percentage, then the project sponsor should be notified immediately.In some instances, senior management and approval are required.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
46/56
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
47/56
The Key Components of a Project Audit Program
Risk Management
This element relates to risks that could adversely effect the completion of the project withine g ven ours an or m ng. s s suc as a r par y ven or go ng ou o us ness,
project staff turnover should be evaluated.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
48/56
The Key Components of a Project Audit Program
Quality Assurance Management
This element relates to the process of ensuring that all deliverables are of required qualityan a a wor as een carr e ou accura e y an o an appropr a e s an ar . songoing effort throughout the development cycle and system use.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
49/56
The Key Components of a Project Audit Program
Procurement Management
This element relates to ensuring that all the necessary resources, equipment, hardware,so ware can e o a ne or e pro ec o mee pro ec ea nes.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
50/56
The Key Components of a Project Audit Program
3rd Party Management
Consideration should be given to look at contract agreements, maintenance agreements,m ng o expec e wor comp e on, resource a oca on, p ys ca oca on o par y.
Evaluation of due diligence work is imperative at the earliest point possible.
If the project team is implementing 3rd party software, you should consider where the sourcecode for the software is stored and if in the contract, we are allowed to obtain the source
rd
2011 Muema Lombe
C
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
51/56
The Key Components of a Project Audit Program
Communication Management
This element relates to formal organizational arrangements which should be adopted toac a e success u exp o a on o n orma on ec no ogy. ese nc u e e mpor ance o
having a project sponsor, the need for informal and formal user involvement, the role of
corporate information and project steering committees, and the advantages of standardizedcommunication between these various parties. Historically, many projects have failedbecause of inade uate communication between MIS and the user de artments
2011 Muema Lombe
Th K C f P j A di P
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
52/56
The Key Components of a Project Audit Program
Interdependency Management
This element relates to other projects that depend on the initial project to succeed or otherpro ec s a mus comp e e pr or o s pro ec so s pro ec w succee . e pro ecteam should be aware of all of these projects and just be figured into the timing and
milestone documents.
2011 Muema Lombe
Th K C t f P j t A dit P
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
53/56
The Key Components of a Project Audit Program
Change Management
This element relates to maintaining version control over project documents relatingspec ca y o e n a pro ec . s wou nvo ve any ocumen s a wou e use osupport the System Development Life Cycle of this project and track the project until
completion.
2011 Muema Lombe
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
54/56
The Key Components of a Project Audit Program
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
55/56
The Key Components of a Project Audit Program
Project Conclusion and Turnover Management
If the project was planned correctly and user involvement has been adequate, the installationo e sys em an e rans er o suppor respons y s ou procee smoo y. s aspecific responsibility of the project director and project manager to satisfy themselves,
before allowing a system to be installed, that: All stages of testing, including acceptance testing, have been completed properly.
Adequate user procedures have been prepared and the users are trained to the required level and want to workwith the system positively.
The MIS department is prepared to run the system.
All required controls and security procedures are in place.
2011 Muema Lombe
Summary
8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011
56/56
Summary
METHODOLOGY: Understand the project managementmethodology deployed at your organization
-the project (e.g. pre-implementation? Post-implementation? Silentparticipant in project team? Other?)
: gree on orma , requency an s r u on oreporting.
RISK: Understand project risk factors. SCOPE: Define the scope of your audit.
EXECUTE: Execute and report on the audit.
2011 Muema Lombe