Project Team Ross Tate City Auditor Stacey Linch Senior Internal Auditor IT Aaron Cateil Internal Auditor Project Number 1190044 This report can be made available in alternate format upon request. Information Technology Services Department IT Governance – Investment, Funding, and Decision Making April 30, 2019 Report Highlights IT Governance Process The City implemented an IT Governance framework similar to industry-recognized standards. However, exception procedures have not been formalized, and performance metrics have not been established. Business Investment Request Review The business investment requests we reviewed were accurate, complete, and contained the required supporting documentation. City Auditor Department 140 N 3 rd Avenue Phoenix, AZ 85003 602-262-6641 (TTY use 7-1-1) Mission Statement To improve the quality of life in Phoenix through efficient delivery of outstanding public services.
Transcript
Project Team
Ross Tate City Auditor
Stacey Linch Senior Internal Auditor IT
Aaron Cateil Internal Auditor
Project Number
1190044
This report can be made available in alternate format upon request.
Information Technology Services Department IT Governance – Investment, Funding, and Decision Making
April 30, 2019
Report Highlights IT Governance Process
The City implemented an IT Governance framework similar to industry-recognized standards. However, exception procedures have not been formalized, and performance metrics have not been established.
Business Investment Request Review
The business investment requests we reviewed were accurate, complete, and contained the required supporting documentation.
City Auditor Department 140 N 3rd Avenue Phoenix, AZ 85003 602-262-6641 (TTY use 7-1-1)
Mission Statement
To improve the quality
of life in Phoenix
through efficient
delivery of outstanding
public services.
Page 2
City Auditor Department
Executive Summary
Purpose Our purpose was to determine if the City’s governance structure for making information system investment decisions followed City policy and Information Technology (IT) standards. In addition, we compared the City’s governance structure to industry standards.
Background The operations of the City and the delivery of services to the public rely on thoughtful, efficient, and innovative investment in technology. Efficient and innovative delivery of technology services depends on a governance structure that defines authority and decision rights, and promotes strategic alignment with the City’s business goals. Information Technology Services (ITS) facilitates the investment, funding, and decision-making processes for technology projects Citywide through the annual technology planning process. ITS works with departments to identify and plan for technology investments that strategically align with the City’s processes and objectives.
Results in Brief Although an overall governance process is in place, performance metrics and key performance indicators still need to be identified, implemented, and monitored.
The City implemented a business investment framework to help departments submit requests in a timely manner and to ensure requests align with the strategic goals of the City. A review of the governance process and associated procedures identified a lack of performance metrics and key indicators. Performance metrics and key performance indicators, such as return on investment (ROI), resource utilization, percentage of canceled projects, and customer satisfaction index, are not defined or implemented. However, the internal tracking system used by ITS was not designed to (1) track the various stages of the process, and (2) identify and alert staff of deadlines or updates to the IT investments and procurements. Overall, business investment requests reviewed were submitted with sufficient documentation.
Overall, the business investment requests reviewed were submitted within the defined timeframe and provided the required information as outlined in applicable policies and procedures. No exceptions were noted.
Page 3
City Auditor Department
Department Responses to Recommendations
Rec. #1.1: Review available metrics and implement key performance indicators that align with the Business Investment Framework and applicable policies and procedures, such as cycle time, actual project cost, cost variance, customer satisfaction, return on investment (ROI), and resource utilization.
Response:
1) ITS will create performance metrics for BIRFs ($8,600+) for technology procurements.
2) ITS will create performance metrics for Board approved technology plans ($750,000+).
Target Date: 07/01/2019
Explanation, Target Date > 90 Days: [Type response here]
Rec. #1.2: Evaluate the current internal tracking system to determine the capability of capturing data to report on the metrics identified in Recommendation 1.1. If it is determined that the current system cannot provide the data needed, evaluate if other tracking systems can provide it.
Response:
Current Business Investment Request Form (BIRF) system capabilities, which are dated to 2015, do not allow for desired or expansive reporting. However, ITS will define requirements for a new application to explore the feasibility, cost, and schedule to replace the current BIRF system.
Target Date: 07/01/2019
Explanation, Target Date > 90 Days: [Type response here]
Rec. #1.3: Develop and publish a BIRF review standard that establishes responsibilities and review timeframes for departments and ITS. Align the review procedures to the review timeframes and responsibilities outlined in the delegate training and BIRF intake and processing materials.
Response:
ITS will create a new Standard Operating Procedure (SOP B.1.5.4) that will serve to establish internal ITS business processes, roles and responsibilities, and other workflow; to create a consistent process for internal review and documentation of processed BIRFs.
Target Date: 07/01/2019
Explanation, Target Date > 90 Days: [Type response here]
Page 4
City Auditor Department
Rec. #1.4: Update IT Standard B.1.5.1 – Business Investment Request Procedure and IT Standard B.1.5.2 – Technology Planning Process & Templates to identify the allowable exceptions to the business investment procedure and the technology planning process, and identify how exceptions will be documented.
Response: ITS will review and revise B.1.5.2., Annual Technology Planning Procedure, to provide departments with clearer information on allowable exceptions to the Annual Technology Planning Program. Criteria has already been included in SOP B1.5.1, Business Investment Request Procedure.
Target Date: 07/01/2019
Explanation, Target Date > 90 Days: [Type response here]
Rec. #1.5: Evaluate the need for a waiver approval for IT Standard B.1.5.1 – Business Investment Request Procedure and IT Standard B.1.5.2 – Technology Planning Process & Templates. If it is determined that a waiver approval is needed, establish a process and document a standard which includes at a minimum:
• required documentation to be submitted by departments with waiver requests,
• a process to evaluate systems to limit risk to the City’s current environment, and
• required levels of approval for waivers.
Response:
ITS will revise SOP B.1.5.1 and B.1.5.2 to reference established IT Waiver Standard (B1.3) and Requesting/Renewing an IT Waiver (B1.3.1).
Target Date: 07/01/2019
Explanation, Target Date > 90 Days: [Type response here]
Rec. #2.1: Ensure requests are complete and contain supporting documentation that justifies review and approval.
Response: ITS has begun new practices that complete this request. ITS regularly sends back BIRFs that are sent for review without required information. Additionally, for requests for the annual technology planning process, ITS works closely with the submitting department to ensure all information is included.
Target Date: Complete
Explanation, Target Date > 90 Days: [Type response here]
Page 5
City Auditor Department
1 – IT Governance Process
Background The Business Investment Request Procedure was established in 2014 to:
• promote transparency of Citywide technology purchases and procurements,
• create a repository for monitoring,
• measure and categorize technology expenditures,
• advocate technical review, and
• align technology expenditures according to standards and procedures. The process applies to all technology goods and/or service acquisitions related to technology, regardless of existing procurement vehicles that are in place. Effective in 2017, procurements exceeding $8,600 require departments to complete a Business Investment Request Form (BIRF) outlining the technology request details. Procurements with a 5-year aggregate cost exceeding $750,000 require departments to complete a technology plan comprised of a business case, financial analysis, and other supporting documentation. The process was adopted and incorporated into the Finance Department’s ProcurePHX review and Annual Procurement Planning processes, and the City’s legislative agenda management system for Council approval. All requests for the procurement of technology must be approved by the CIO; some requests may require approval by the Business Investment Board based on the CIO’s recommendation. An overview of the business investment request approval process is listed in Attachment B.
Results The City implemented an IT Governance framework similar to industry recognized standards.
The City implemented a business investment framework to help departments submit requests in a timely manner, and to ensure technology projects align with the strategic goals of the City. Through the development of the framework, ITS established Administrative Regulation 1.94 – Business Investment Framework, Information Technology Services Standard Operating Procedure B.1.5.1 – Business Investment Request, and Standard Operating Procedure B1.5.2 – Technology Planning Process & Templates, which provided departments with the guidelines and procedures related to technology investment requests. The Control Objectives for Information and Related Technologies 5 framework (COBIT 5) is a widely adopted business framework developed by the Information Systems Audit
Page 6
City Auditor Department
and Control Association (ISACA) for the governance and management of enterprise IT. Through our interviews with staff, reviews of City policy, and reviews of supporting documentation, we identified how the City’s business investment framework aligned to the COBIT 5 framework. (See diagram on the following page.) In general, the City’s framework aligned with the five principles of the COBIT 5 framework. No exceptions were noted.
Page 7
City Auditor Department
COBIT 5 Framework principles compared to City’s Business Investment Framework
Overall, the City’s framework aligned with the COBIT 5 framework.
Principle 1 - Meeting Stakeholder Needs
•City Departments - submit requests with adequate lead time for review and recommendations
•Chief Information Technology Officer (CIO) - reviews business requests and selects requests to be further reviewed by the Board and Ad Hoc Tech Commitee
•Board (may include public members/or community leaders) -approves business requests and determines funding
•Ad Hoc Tech Committee - performs and presents business and financial analysis of requests that require Board review
Principle 2 - Covering the Enterprise End-
to-End
•ITS Standard Operating Procedure - b1.5.1 - Business Investment Request Procedure - provides guidance on submitting requests and defines roles and responsiblities of parties throughout the Enterprise
Principle 3 - Applying a Single, Integrated
Framework
•Administrative Regulation 1.94 - Business Investment Framework - framework that determines technoogy investments, policies, and strategies in support of the City's business needs
Principle 4 - Enabling a Holistic Approach
•Administrative Regulation 1.94 - Business Investment Framework - framework that determines technology investment policies and strategies in support of the City's business needs
• ITS Standard Operating Procedure - b1.5.1 - Business Investment Request Procedure - provides guidance on submitting requests and defines roles and responsiblities of parties throughout the Enterprise
•Business Investment Request Form - A formal request submitted to ITS for planning and/or procuring technology goods or services
Principle 5 -Separating
Governance from Management
•Governance - CIO authorization for technology purchases and Board determines funding and resources
•Management - ITS Enterprise Architecture Team works with departments to ensure requests are complete and monitored throughout process. Technology Domain Owners Delegates ensure staff are involved and requests are aligned with standards
Page 8
City Auditor Department
The Business Investment Framework can be improved though additional training and communication.
We interviewed staff from the Human Resources and Public Transit Departments as well as members of the Business Investment Board (Board) to obtain a better understanding of the process, how the process has changed, and to identify areas that require improvement. Departments and other stakeholders identified performance, training, communication, and transparency as areas the governance process could improve. The information obtained is summarized below: Performance
o Develop metrics to gauge performance and measure project development along different timeframes.
o Track previously submitted projects by departments to identify department history and capability to handle larger projects/investments.
o Implement notifications in the tracking system to allow for status updates for both departments and ITS.
o Remove requests that have not been approved or submitted from internal tracking system.
o Review how turnover of project managers and loss of additional resources negatively impact projects.
Training
o Provide additional training regarding technology plans as processes are updated to department liaisons.
o Improve training to help departments prepare for presentations and provide sufficient justification for their business needs.
Communication
o Provide updates and statuses of approved projects which include enough detail and information regarding conditional approvals.
o Develop a survey of departments and incorporate feedback as needed.
o Develop a workflow that incorporates performance metrics. Transparency
o Define a standard timeframe for updates (e.g., requests will be reviewed in 10 business days)
ITS should establish key performance indicators and report them to the Board and City Management regularly.
ITS utilizes an internal database to track investment requests submitted by departments. We analyzed the database and found that 2,763 requests were submitted since 2015; of which, 2,310 were approved. The number of days between the request
Page 9
City Auditor Department
creation and the approval averaged between 16 and 53 days. A summary of our results is listed in the table below:
Business Investment Requests 2015-2019
Approximately 83% of requests have been approved, with most requests being approved in 2017.
Although ITS is tracking investment requests, the system currently being utilized is not able to track performance metrics and other key performance indicators that would provide transparency at a departmental and governance level. Indicators and metrics, such as approval time (time between request submittal and approval), actual project cost (project-related expenses used to date), cost variance (planned budget vs. actual budget), and return on investment (ROI) are not being tracked within the system. The system currently tracks all requests regardless of status or age, and includes requests that have been submitted as a revised request. ITS will need to determine if older or duplicate requests should be deleted in order to obtain accurate metrics. ITS has not documented all exceptions to the business investment procedure or technology planning process. In addition, a waiver process has not been established.
IT Standard B.1.5.1 – Business Investment Request requires an approved BIRF prior to Council approval, and/or the purchase order being issued, to ensure alignment with the City’s overall strategy for investing in and/or continuing to invest in technology. Requests are reviewed by the Enterprise Architecture Team and can be approved by City leadership designated by the CIO. The CIO determines which requests require Board approval. The Business Investment Framework and supporting IT Standards apply to all departments; however, we noted some technology investments that did not follow the established procedures, including technology investments made by ITS. ITS stated that in certain circumstances a request will not follow the investment request process based
2,310
11
21653 7
166
Approved Deferred In process Not Approved Pending BoardReview
PendingSubmission
0
500
1,000
1,500
2,000
2,500
Page 10
City Auditor Department
on decisions made by executive management. For example, if the project is considered as part of an operational budgeting procedure, or if the project has a high-profile scope. ITS stated its investments were reviewed directly with executive management for approval rather than requiring Board approval. Procedures were not in place to document the approval of requests that did not follow the process. ITS stated departments have requested to be exempt from the investment request process, as their systems are exclusively supported by their respective departments. ITS is currently reviewing the implementation of a waiver approval process for requests to be exempt from the process. ITS will need to ensure the waiver approval processes define the following:
• the supporting documentation to be provided by departments, and
• the process to evaluate technology to determine: (1) the potential allocation or commitment of ITS staff resources, (2) the impact to current systems or technology architecture, (3) the potential impact to network security, (4) or the impact based on previously identified compliance factors.
Recommendations 1.1 Review available metrics and implement key performance indicators that align with
the Business Investment Framework and applicable policies and procedures, such as cycle time, actual project cost, cost variance, customer satisfaction, return on investment (ROI), and resource utilization.
1.2 Evaluate the current internal tracking system to determine the capability of
capturing data to report on the metrics identified in Recommendation 1.1. If it is determined that the current system cannot provide the data needed, evaluate if other tracking systems can provide it.
1.3 Develop and publish a BIRF review standard that establishes responsibilities and
review timeframes for departments and ITS. Align the review procedures to the review timeframes and responsibilities outlined in the delegate training and BIRF intake and processing materials.
1.4 Update IT Standard B.1.5.1 – Business Investment Request Procedure and IT
Standard B.1.5.2 – Technology Planning Process & Templates to identify the allowable exceptions to the business investment procedure and the technology planning process, and identify how exceptions will be documented.
1.5 Evaluate the need for a waiver approval for IT Standard B.1.5.1 – Business
Investment Request Procedure and IT Standard B.1.5.2 – Technology Planning Process & Templates. If it is determined that a waiver approval is needed, establish a process and document a standard which includes at a minimum:
• required documentation to be submitted by departments with waiver requests,
Page 11
City Auditor Department
• a process to evaluate systems to limit risk to the City’s current environment, and
• required levels of approval for waivers.
Page 12
City Auditor Department
2 –Business Investment Requests Review
Background City Administrative Regulations (ARs) and IT standards guide departments in submitting a business investment request. Attachment A, City Technology-Related Policies, summarizes these guidelines. The Business Investment Request Form is a formal request submitted to ITS for planning and/or procuring technology goods and services. Business investment requests for purchases greater $8,600 require a Business Investment Request Form and CIO authorization. Requests that are greater than or equal to $750,000 are defined as technology plans and are subject to the following requirements:
• an architecture review (governance body that reviews requests to evaluate architectural impact),
• a presentation to the Board (Executive-level governance body that makes decisions on technology plans and/or tech-spend requests by the CIO), and
• supporting documentation in the form of a business case and financial analysis worksheet.
To assess compliance, we reviewed the business investment framework, applicable business investment request policies and standards, submitted business investment request forms, and supporting documentation.
Results Overall, the requests submitted were compliant with the applicable policies and procedures.
During fiscal years 18/19 and 19/20, twenty IT investment requests were submitted as tech plans for review. We reviewed a sample of four requests, including supporting documentation, to determine if the requests followed A.R 1.94 – Business Investment Framework, ITS SOP B.1.5.1 – Business Investment Request, and ITS B1.5.2 – Technology Planning Process & Templates. Most of the requests were accurate, complete, and contained supplemental support documentation. Below is a summary and status of the projects reviewed.
Page 13
City Auditor Department
IT Investment Projects Submitted
All projects were presented to the Business Investment Board. The Human Resources Department submitted a Technology Plan for a Learning Management System to track and manage training. The original project start date requested was 7/31/2017 with a completion date of 12/31/2018. The project was recommended for approval by the Board and is under consideration for budget appropriation for FY19/20. The Human Resources Department submitted a Technology Plan for a Citywide Time and Labor System to track employee work schedules and payroll. The original project start date requested was 7/01/2017 with a completion date of 12/31/2018. The project implementation was delayed due to missing supplemental documentation from the Human Resources Department and time needed to conduct additional research to support justification. Neighborhood Services submitted a Technology Plan for phase two of its Rehab CMS Replacement project. Initially, this request was part of FY18/19; however, the department postponed the project. The project was requested to start on 7/1/2019 and be completed on 12/31/2020. The project was recommended for approval by the Board and is under consideration for budget appropriation for FY19/20. The City Clerk Department submitted a Technology Plan for its Automated Ballot Tabulation project to acquire a new system for processing election ballots. The project request was originally to start on 7/1/2016 and to be completed on 12/31/2018; however, the City Clerk Department has delayed the project to handle the unscheduled elections occurring in 2019. The project is currently in the BIRF creation phase and has not be approved by the CIO and the Board.
• City-wide eLearning platform to track and manage training• Initial Cost: $332,000• Annual Maintenance Cost: $232,000
Human Resources Learning Management System
• City-wide application to manage data and employee payroll• Initial Cost: $1,080,900• Annual Maintenance Cost: $53,900
• Ballot sorting system to automate election tasks• Initial Cost: $225,000• Annual Maintenance Cost: $25,000
City Clerk Ballot Tabulation
Page 14
City Auditor Department
Recommendations 2.1 Ensure requests are complete, and contain supporting documentation that justifies
review and approval.
Page 15
City Auditor Department
Attachment A – Related City Policies and Procedures A.R. 1.94 – Business Investment Framework
A.R. 1.94 defines the City’s management of technology that is governed by the Business Investment Framework. This framework consists of three components working together to ensure technology investments are supporting the City’s business goals. The framework consists of the Chief Information Officer (CIO), the Business Investment Board (Board), and the Ad Hoc Technology Committee(s). ITS SOP b1.5.1 – Business Investment Request
The Business Investment Request Procedure was established in 2014 to promote transparency of Citywide technology purchases and procurements, to create a repository for monitoring, to measure and categorize technology expenditures, to advocate technical review, and to align technology expenditures with standards and procedures. ITS SOP b1.5.2 – Technology Planning Process & Templates
Annual technology planning is a process facilitated by Information Technology Services (ITS) to proactively partner with departments to: identify, plan for, and prepare written business cases; prepare financial analysis; assess strategic alignment; and identify technology architecture. This process should be utilized whether proposed projects have a budget allocation or not. Submitted technology planning requests are reviewed and processed by ITS, and are approved by City leadership based on strategic alignment with the City’s objectives. ITS SOP b1.5.6 – Business Investment Board Presentation Procedure
This procedure was created to provide guidance for individuals who will be presenting business investment request(s) to the Business Investment Board (Board). The target audience for this guideline is middle management staff who are considered a sponsor or champion of a business investment request that has been elevated to the Board for its review, consideration, and decision. One of the overall goals of this procedure is to empower each department to exercise a level of creative control over its presentation, which the Board members will be considering.
Page 16
City Auditor Department
Attachment B – BIRF Approval Process
Page 17
City Auditor Department
Scope, Methods, and Standards
Scope We evaluated business investments requests presented at the October 2018 business investment board meeting and the applicable policies and procedures that govern the overall request process. We conducted our testing in December 2018 and January 2019.
Methods We used the following methods to complete this audit:
We evaluated compliance with City policies by reviewing procedures, practices, and business investment requests related to technology investments. We gained a basic understanding of the current process by reviewing requests. These requests detail what the business needs are, the proposed benefits and values, financial sources, and associated costs. We met with staff from various departments and members of the Board to obtain additional insight into the request process, and we identified areas that require improvement. Unless otherwise stated in the report, all sampling in this audit was conducted using a judgmental methodology to maximize efficiency based on auditor knowledge of the population being tested. As such, sample results cannot be extrapolated to the entire population and are limited to a discussion of only those items reviewed.
Standards We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the performance audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
