State of Hawaii Department of Accounting and General ......HawaiiPay Project Deployment Audit Report...

621 Capitol Mall, Suite 1425

Sacramento, CA 95814 916-565-8090

www.publicconsultinggroup.com

State of Hawaii

Department of Accounting and General Services (DAGS)

Office of Enterprise Technology Services (ETS)

HawaiiPay Project

IV&V Deployment Audit Report – Group 2

Version 1.1 - Final

August 23, 2018

HawaiiPay Project Deployment Audit Report – Group 2

Final – August 23, 2018

Document History Page i

Document History

Version Date Brief Description of Modifications

1.0 August 5, 2018 Draft submitted to state for review.

1.1 August 23, 2018 Finalized deliverable submitted to state.

Document Author & Contact Information

Name Title Contact Information

Michael Fors PCG Project Manager [email protected]

Ken Wilmoth PCG HCM / ERP SME [email protected]

Traci Veteto PCG PM SME [email protected]

mailto:[email protected]





Table of Contents Page ii

TABLE OF CONTENTS

1 INTRODUCTION ................................................................................................................4

1.1 Purpose ................................................................................................................................ 4

1.2 Background .......................................................................................................................... 4

1.3 Summary .............................................................................................................................. 5

2 APPROACH.......................................................................................................................6

2.1 Analysis Approach ................................................................................................................ 6

2.2 Terms and Definitions ........................................................................................................... 8

3 ANALYSIS .......................................................................................................................10

3.1 Adherence to best practices ................................................................................................ 10

Testing ............................................................................................................................... 10

System Integration .............................................................................................................. 11

System Change Management ............................................................................................. 11

Organizational Change Management (OCM) ....................................................................... 12

3.2 Satisfaction of appropriate requirements ............................................................................. 13

Testing ............................................................................................................................... 13



3.3 Proper system functionality ................................................................................................. 14

Testing ............................................................................................................................... 14


3.4 Ability to support program business needs .......................................................................... 15

Testing ............................................................................................................................... 15



Organizational Change Management .................................................................................. 16

Help Desk ........................................................................................................................... 16

4 FINDINGS & RECOMMENDATIONS...............................................................................17

APPENDIX A: IV&V FINDINGS AND RATINGS DEFINED ......................................................19

APPENDIX B: ASSESSMENT CATEGORY DEFINITIONS AND RATINGS ............................23



Table of Contents Page iii

Table of Tables Table 1: Common Terms ......................................................................................................................... 8

Table 2: Risk Rating Matrix .................................................................................................................... 20

Table 3: Risk Rating Definitions ............................................................................................................. 21

Table 4: Issue Rating Definitions............................................................................................................ 21

Table 5: Assessment Category Definitions ............................................................................................. 23

Table 6: Assessment Category Rating Matrix ......................................................................................... 26

Table of Figures Figure 1: Eclipse IV&V® Technical Assessment Methodology .................................................................. 7



INTRODUCTION Page 4

1 INTRODUCTION

The State of Hawaii’s (SOH) Office of the Enterprise Technology Services (ETS) acquired the services of the Public Consulting Group – Pacific Point (PCG-PP), hereafter referred to as PCG, to provide Independent Verification and Validation (IV&V) services for the HawaiiPay Project with the Department of Accounting and General Services (DAGS). These services include ongoing periodic assessment and monthly reports. IV&V reports are intended to describe key activities, current status, any findings or concerns, as well as an independent perspective of the project’s current state of risk.

This report describes IV&V’s assessment of the HawaiiPay Project’s system preparedness prior to the production implementation of Deployment Group 2 with a special focus on adherence to best practices and requirements, functionality and the ability to support program business needs. Since the project has been preparing for the implementation of Group 2, IV&V’s approach aimed at being as non-intrusive as possible in order to avoid disruptions to the project. This section summarizes IV&V’s preliminary concerns, risks, and/or issues resulting from this assessment. Specific details supporting each finding are elaborated in Section 3, Analysis and Findings.

1.1 Purpose

This report describes IV&V’s assessment of the deployment activities for Group 1 and Group 2 of the HawaiiPay project including the current state of the project’s risk. This assessment focuses on the specific project activities as outlined in the IV&V Independent Verification and Validation Plan (IVVP). The information provided here is intended to be informative but also succinct. The sections herein summarize activities and highlights findings related to the deployment activities identified by the PCG IV&V team. Specific description of the findings can be found below in the Findings section. A more comprehensive narrative regarding the specific activities can be found below in the Analysis and Findings Section.

1.2 Background

The HawaiiPay Project is a statewide initiative intended to modernize the current Payroll system into one integrated statewide solution. The state contracted with a system integrator (CherryRoad) to provide key management and technical services for the duration of the HawaiiPay Project. To provide the required functionality, the state chose PeopleSoft, an established commercially available off the shelf (COTS) solution. An existing instance of PeopleSoft has already been deployed for Department of Human Resources Development (DHRD). The state chose to utilize this existing instance to support all state employees.

As noted in the IV&V Initial Assessment Report, the DHRD instance was relocated to a commercial data center. As part of the system integration contract, CherryRoad was engaged to relocate the DHRD instance and assume operations of the PeopleSoft application for all Human Capital Management (HCM) functionality. CherryRoad



INTRODUCTION Page 5

completed the transition and assumed operations of the PeopleSoft application prior to PCG IV&V contract. At the time of this assessment, IV&V is not aware of any known severe or critical issues related to the operations and support of the relocated PeopleSoft application. In addition to moving operations, CherryRoad successfully implemented a Disaster Recovery Data Center, and has performed multiple Disaster Recovery tests. There have been no severe or critical issues reported since CherryRoad took over operations in May 2017.

The HawaiiPay project went live with its first population of end users defined as Group 1 in April/May 2018, a small pilot of departments, and payroll functionality. As of the publication of this report the project is currently in the cutover process for going live with Group 2. Group 2 has a much larger user population and includes issues not present with the Group 1 population. Group 2 is the second of 3 planned groups that cover the entire Hawaii state employee population.

1.3 Summary

During the review of the deployment activities, IV&V did not discover any critical concerns with the project’s processes, methods, tools, procedures or leadership. IV&V found the general state of readiness to deliver the HawaiiPay service, to be adequate. However, IV&V did note a number of opportunities for the project to improve its efficiencies and reduce risk. A number of these findings were related to the lack of “Best Practice”. IV&V is well aware that not all generally accepted best practices are required for a successful project, however the implementation and continual use of these practices can help boost confidence in the project’s ability to meet its goals, timelines and budget. IV&V also noted that a number of key processes are highly dependent on specific individuals which may add unnecessary risk if staffing changes became necessary in key project areas. Some of the key concerns and other more positive findings are outlined below and include:

• IV&V noted that the processes, communication strategies and incentives used to coordinate the number of employees who sign up for direct deposit via the ACH process has been more effective for the Group 2 population. As the submission of this report, the population of Group 2 employees that registered for direct deposit, has exceeded project leadership expectations.

• IV&V noted that significant improvements to the processes used to ensure that third party interfaces were being developed in a timely manner and tested in accordance with the projects testing protocols.

• IV&V noted that a number of key processes are manual and somewhat informal. Again, this may be adding unnecessary risk, for example, to protect the production environment, a process to prevent unapproved, untested or unnecessary changes from being deployed in the production environment is necessary. The HawaiiPay project depends on a specific technical resource to ensure this does not occur. A best practice may include the deployment of tools that require very specific steps to ensure that unwanted changes are not deployed. This risk is exacerbated by the fact that there is more than one channel of system changes that must be managed. The project has a development channel and so does the DRHD team and the



Approach Page 6

production code base is shared between both teams. This increases the risk of changes from one team overlaying the changes from the other team. A diligent effort must be undertaken to ensure this does not negatively affect the production environment.

• IV&V has identified risks that could reduce the effectiveness of OCM efforts including lack of sufficient resources, overtaxed resources, and the lack of a dedicated OCM strategic lead. OCM communications to banks have not always produced their intended result and inaccurate instructions have been sent to state employees by the banks.

• IV&V also noted a lack of significant regression testing. Regression testing is another method to help protect a production application. IV&V noted a number of small changes were approved for deployment into the production environment without a subsequent regression testing cycle.

• IV&V also noted a lack of evidence to demonstrate that system, integration and user acceptance testing was completed successfully. In the absence of specific documentation (print screens, etc.) IV&V cannot validate that the testing was executed correctly, and the results were as expected.

• IV&V also noted that elements of system defects are tracked (duplicated) in multiple systems (Implementation Tracker and ServiceCloud). When an issue is reported to the Help Desk and determined to be a defect, the resolution of the defect is then tracked in the Implementation Tracker. While defects found through other sources may not be entered into the HawaiiPay help desk tool (ServiceCloud), but entered directly into Implementation Tracker. This may be a redundant process and leaves the project without a single reporting source for all reported issues and lacks good traceability.

2 APPROACH

2.1 Analysis Approach

The PCG IV&V team utilizes the Eclipse IV&V® Technical Assessment Methodology, depicted in Figure 1, to establish and deliver IV&V findings throughout all IV&V work products. Executing the tasks using this common methodology helps ensure that all pertinent facts are gathered, the relevant stakeholders are consulted, there is a clear understanding about any findings resultant from the assessment, and that the assessment report is objective, accurate and does not result in surprises to stakeholders.



Approach Page 7

Figure 1: Eclipse IV&V® Technical Assessment Methodology

The Eclipse IV&V® Technical Assessment Methodology includes four primary actions:

• Discovery — the IV&V team reviews project documentation, work products, deliverables, along with any plans or schedules that apply. The IV&V team interviews key project team members to gain a thorough understanding of the assessment area, identifying applicable standards, best practices, and lessons learned to be used as evaluation criteria.

• Research and Analysis — the IV&V team conducts research and analysis of specific aspects of the component or process being assessed in order to form an evaluation of the validity of the approach. Once the initial analysis is completed, the assessment preliminary results are documented for clarification.

• Clarification — the IV&V team seeks clarification, as needed, from key project team members on aspects of the organization and communication processes to ensure agreement and concurrence on the results of the discovery, research, and analysis.

• Delivery of Findings — the IV&V team’s assessment and status reports document the results of discovery, research, analysis, and clarification, presenting detailed findings and documentation of project strengths. These reports contain measurement dashboards, observations/findings, risk assessments, and risk mitigation strategies. Before the delivery of findings, they are reviewed internally by IV&V team members, so that any gaps or inconsistencies can be identified and corrected.

For this report, IV&V conducted informal interviews with various members of the HawaiiPay project team and stakeholders.

Industry Standards and Best Practices



Approach Page 8

PCG applies and abides by best practices in the information technology industry, including, but not limited to, standards and methodologies issued by:

• Institute of Electrical and Electronics Engineers (IEEE)

• The Project Management Institute’s (PMI), Project Management Book of Knowledge (PMBOK)

• Information Technology Infrastructure Library (ITIL)

• International Organization for Standardization (ISO) 9000

• National Institute of Standards and Technology (NIST)

• Center for Internet Security (CIS)

2.2 Terms and Definitions

This section contains a list of terms (i.e., abbreviations, acronyms, and notations) used in this assessment and their definitions to provide a common understanding.

Table 1: Common Terms

Term Definition

ALM Application Lifecycle Management

BAFO Best and Final Offer

CIS Center for Internet Security

COI Communities of interest

COTS Customer off-the-shelf

DAGS Department of Accounting and General Services

DHRD Department of Human Resource Development

ERP Enterprise Resource Planning

ETS Office of Enterprise Technology Services

HCM Human Capital Management

IEEE Institute of Electrical and Electronics Engineers

ISO International Organization for Standardization

ITIL Information Technology Infrastructure Library



Approach Page 9

Term Definition

IV&V Independent Verification and Validation

HawaiiPay HawaiiPay Project

M&O Maintenance and Operations

NIST National Institute of Standards and Technology

OCM Organizational Change Management

PCG Public Consulting Group

PII Personal identifying information

PMBOK Project Management Body of Knowledge

PMI Project Management Institute

PMP Project Management Plan

PP Pacific Point

RIOD Risks-Issues-Opportunities-Decisions

RTM Requirements Traceability Matrix

SDLC Software Development Life Cycle

SME Subject Matter Expert

SOH State of Hawaii

TPA Third party administrator



Analysis Page 10

3 ANALYSIS

This section includes summaries of the analysis associated with IV&V findings by category. IV&V rating definitions for findings can be found in Appendix A.

3.1 Adherence to best practices

This section will describe how the processes, procedures, plans and strategies match up against specific standards and generally accepted best practices.

Testing

As outlined in the previous IV&V Project Initial Assessment report, best practices for testing outlined in the IEEE SDLC standards, requires multiple levels of testing. The specific testing cycle is dependent on what is being tested. A description of these common testing cycles is below:

1. Unit Testing – testing for custom extensions (customizations) and configurations starts with “Unit” testing by the functional or technical person responsible for the development of the custom object or configuration change.

2. System Testing – System testing for custom extensions and configurations is required once the Unit Testing is complete. System testing often utilizes more comprehensive testing scripts that are designed to meet the objectives of the testing scenarios described in the functional and technical specifications.

3. User Acceptance Testing – Once all the defects discovered during System Testing have been resolved and re-tested, new custom extensions and configurations are often tested by end users as part of the general acceptance of the solution.

Other testing cycles are often employed to ensure that specific elements are tested. These additional testing cycles may include:

• Integration Testing – When new or modified objects or configurations are deployed to an existing solution, the solution is often tested to ensure the new objects or configurations do not cause unexpected issues in other areas of the solution.

• Performance Testing – To help ensure that new or modified solutions meet the organizations service level objectives and availability expectations, these solutions are often put through a stress test that emulates a number of end users greater than the expected number of concurrent users in the production systems. Specific transactions that represent the most resource intensive functional areas are often selected for performance testing.

• Parallel Testing – When deploying new business applications, Parallel Testing is often performed to compare results between specific transactions previously executed in the legacy system with the results of similar transactions in the new solution. The results of Parallel Testing include records or groups of records that match the legacy solution along with “known” and expected differences. Any other variance could be considered a defect and requires a resolution before the new



Analysis Page 11

solution can be deployed. Parallel Testing is often utilized during data conversion testing to compare the data records between the legacy system and the new solution to ensure the data extraction, transformation and migration was completed successfully.

IV&V noted that as a matter of practice, the project follows these basic testing standards. The project executes testing cycles that mostly conform with the best practices outlined above. However, some concerns regarding the ability to validate testing results are noted below in the Findings section.

System Integration

A long standing and common practice when implementing Enterprise solutions that require integration with third parties, is to construct a mechanism that extracts a specific data set, then formats that data to meet an agreed standard. This is commonly done using an American Standard Code for Information Interchange (ASCII) comma separated text file. This process is utilized by both inbound and outbound interface partners.

As a matter of best practice, IV&V noted that more contemporary mechanisms for system integration such as Service Oriented Architecture (SOA) utilizing a Service Oriented Architecture Procedure (SOAP) request sent via a dynamic Webservice have become common practice. These types of technologies may be more efficient and could be used at any time for one or more integration transactions instead of sending a single file for processing once a day. However, the development and deployment of these technologies may require expertise and a toolset that all interface partners may not have. Deploying Webservices for all system integration transactions can be challenging in some legacy systems. To overcome these challenges, completely new systems and system development tools may need to be deployed. Given the successful track record of using ASCII text files transported via Secure File Transport Protocols (SFTP), the project decided to continue to use these methods to integrate all third-party systems.

IV&V noted that there are potential security concerns when sending files to external third parties via SFTP. Though the project has some security controls in place (individual SFTP credentials instead of shared functional accounts, separate in/out folders for each entity) IV&V recommends that controls be put in place to ensure that unnecessary access to HawaiiPay interface files is prevented. These controls should include:

• The archiving and removal of all interface files from the SFTP servers as part of the interface processing

• Clearing any data staging areas of inbound interface files as soon as the interface data has been processed

• Following industry best practices regarding password control for SFTP services

System Change Management

Contemporary Enterprise solutions often utilize the standards and practices outlined in the Information Technology Infrastructure Library (ITIL) Service Management guidelines,



Analysis Page 12

to establish processes, procedures and management decision guidelines to control how production systems are updated. Using a structured repeatable process to make updates to the production environment helps prevent system unexpected service downtime. The ITIL Service Management standards include:

• Change Control Board (CCB) – A council that reviews all planned and emergency production system change requests. The CCB reviews and approves the change request prior to the development of any system changes. Then again approves the changes once all the testing cycles have been completed prior to being “pushed” into the production environments.

• Toolsets to ensure system changes don’t overlay other changes – These tools flag changes that require objects that may be part of other approved changes.

• Limited access to production systems and configurations – Controlled access to both the system objects and the configuration should only be granted to key support personnel. All testing and troubleshooting by functional or technical resources should be completed in other environments.

• Specific testing cycles (including regression testing) – Full system integration testing is necessary for any new changes to the production environment. New changes should be tested to both ensure they function as designed but also do not disrupt other parts of the system or service.

IV&V noted that some of these expected processes are in place in the HawaiiPay project, but may lack the formal structure commonly seen in enterprise projects. For instance, the project does have a Change Control mechanism, however, change requests may not be fully regression tested prior to being pushed to production. The project also has a manual process to manage production updates, however the project must also be cognizant of changes requested by another production support channel at DHRD. Having multiple channels for system changes requires more complex management of change requests. IV&V recommends that the project establish a more detailed system integration testing mechanism. The SI has recommended DHRD changes go through HawaiiPay change control.

Organizational Change Management (OCM)

IV&V has noted several instances of effective OCM execution. IV&V has also identified some risks that could reduce the effectiveness of OCM efforts including lack of sufficient resources, overtaxed resources, and the lack of a dedicated OCM strategic lead. For example, despite multiple bank communications, IV&V was informed that American Savings Bank (ASB) had recently sent a letter to all their state employee members describing HawaiiPay changes that were only applicable to Group 2 employees. This is the second known instance where a financial institution sent inaccurate communication to their members. IV&V has recommended HawaiiPay take additional steps to clarify important messages and that the project provide more overt, simplified, clarifying details/instructions (especially for stakeholders who may misunderstand or misconstrue messages/instructions) as well as improved follow through to validate of communications. Further, IV&V recommends HawaiiPay request TPA’s forward all draft



Analysis Page 13

planned communications with regard to HawaiiPay to the project for review prior to sending.

3.2 Satisfaction of appropriate requirements

Testing

The project uses a custom toolset (Implementation Tracker), developed in PeopleSoft to track key project activities. These activities include:

• Requirements Tracing

• Test Case Development

• Test Case Execution Results

• Interface Development and Testing

• Incident/Defect tracking

• System Change Management Activities

All the projects functional requirements are documented in Implementation Tracker. Each requirement is traced throughout the project’s lifecycle. Tests cases for each testing cycle are developed for each requirement and documented in the implementation tracking tool. Test cases are associated with the appropriate requirement. When the test cases are executed, the results are noted in the implementation tracking tool. IV&V noted that as a matter of best practice, the project should include clear documentation or other evidence that each step of a test case was fully executed and passed. Currently the project simply notes that the test case passed or failed. If a step in the test failed, print screens of the results or any errors are attached to the test case. This is used to assist the person who is triaging the defect. In the absence of documented evidence that the test case was fully and successfully executed, IV&V is not able to verify the testing results. However, it should be noted that the number of defects created based on new or modified Reports, Interfaces, Configurations, Enhancements and Workflows (RICEW) objects have been reported at or below an acceptable level.

System Integration

For interface testing, where possible, the project uses the same testing methodology noted above. A significant difference when testing interfaces, is that some third-party testing may be required. This testing is often out of the hands of the project team. IV&V has noted that the project team has implemented improved controls to ensure that the third-party testing meets the testing protocols agreed to by both the project and the third-party.


Requirements are managed via the testing processes outlined above. Once changes are approved to be pushed into the production environment the Change Control Board would have reviewed the change request to ensure that the requirements have been met by the



Analysis Page 14

change. The Change Management processes are the final control opportunity to ensure that requested changes meet the objectives and do not cause issues with other system functionality.

3.3 Proper system functionality

Testing

As noted above, the project executes key testing cycles to help ensure that the new or modified RICEW objects and configurations not only meet the objective of the documented requirement, but also provide the required system functionality. This distinction may not seem critical, but often documented requirements may perform as expected but not be comprehensive enough to ensure the solution is providing the value intended.

System Integration

When integrating two or more distinct systems, it is not always possible to define and document specific requirements prior to developing the interface and data formats. For outbound interfaces, a list of data elements and a specific format for each data element is produced. This high-level design is then sent to the interface partner for validation. This process is iterative and may not be completed until the interface file can be fully processed by the interface partner. To ensure all the system functionality is known and addressed, both the inbound and outbound interface partner must test the interface file as part of an end-to-end business process test. IV&V noted that in some cases for Group 1, this level of information exchange and interface testing may not have been completed. However, as described below, a number of significant changes were made to the interface development and testing processes prior to Group 2. These changes appear to have helped the project better understand each interface requirement and ensure the appropriate effort was being used to prove the interface was meeting the overall system functional requirements. However, IV&V had logged risks with regard to constrained and overtaxed key HawaiiPay project members including the projects only 2 mainframe interface programmers. This risk was realized recently when last minute interface requirements had to be quickly developed in response to the Supreme Court Janus case decision allowing employees to opt-out of union due deductions; mistakes were made, and incorrect paychecks were generated. IV&V recommends reviewing existing data validation processes and procedures (automated and otherwise) to identify opportunities for improvement and implement improvements immediately. Use of enhanced automated data validation (minimally, additional interface validation queries) support could increase data accuracy and reduce time consuming manual processes (see risk #7, “High volume of manual processes at cutover”) for already constrained project resources. Finally, IV&V recommends the project explore the feasibility of having the agencies and TPA's validate the final state payroll data prior to payroll being finalized and run.



Analysis Page 15

3.4 Ability to support program business needs

Testing

IV&V noted that the testing program, as executed for Groups 1 and 2, appear to be adequate to support the program requirements. Testing can never be overdone, but, as a practical matter, decisions must be made regarding what level of detailed testing is sufficient. The project team has defined clear exit criteria for key testing cycles and uses this threshold as a determining factor when deciding to close a testing cycle.

IV&V also noted that Parallel Testing for Group two was completed just prior to the submission of this report. IV&V reviewed both the tools and processes used to collect raw data and generate the parallel testing results. IV&V noted that the tools sets, and process used to generate the results, may be overly cumbersome, but appeared to be adequate to the task. IV&V also noted that the process is highly dependent on one key CherryRoad resource. IV&V was assured that the absence of this key resources could be mitigated by other CherryRoad resources who have a basic understand of the tools and processes used to generate parallel testing results.

IV&V remains concerned that the number of defects discovered during Parallel Testing, that require a manual solution may become overwhelming during the production cutover. To mitigate this concern, IV&V recommends that each task be fully documented, resourced and tracked in the project’s cutover plan.

Other than the concerns noted above, IV&V did not note any significant concerns related to the project’s testing methodologies or results.

System Integration

The HawaiiPay solution’s ability to support the needs of the business not only requires custom logic and configuration, but also requires integration with other state departments and agencies as well as other third-party organizations. During an initial assessment of the HawaiiPay project’s processes and practices, IV&V noted that an adequate documented formal process with interface partners was lacking. A process, that includes the leadership and technical resources for all interface partners to ensure the success of each system integration point is necessary. This process should establish:

✓ What needs to be done

✓ The timeline of when an interface needs to be developed and tested

✓ The systems required for testing

✓ Who will execute each part of the interface development and tests

✓ Who each party should communicate with for each step of the interface development and testing



Analysis Page 16

IV&V also noted that since the submission of the IV&V Project Initial Assessment report, a number of changes have occurred regarding interface development and testing processes. These changes include:

• Established key resources for each interface partner

• Established communication and reporting procedures for each system integration point

• Worked more closely with each interface partner to help ensure that quality and timeline expectations were met

The development and deployment of these new or updated practices has resulted in a higher success rate for interface implementation.


As noted above, complete integration testing and review of the testing results by the testing team are final steps to ensure that the requested changes meet needs of the program.

Organizational Change Management

Effective Organizational Change Management (OCM) addresses the needs of the business prior to system implementation to ensure a successful launch. Overall the project has made great progress in meeting business needs through their OCM efforts. Communication kits have been prepared and sent out to each department to assist in preparing their people for their transition to HIP payroll. Training of agency payroll and HR users has been carefully planned, executed, and tracked to ensure users are effectively trained to operate the system at go-live. Functional leads leveraged User Acceptance Testing sessions to work with agency payroll and HR users to ensure proper understanding of the system and to address any concerns and special circumstances users faced.

One of the primary success factors for HawaiiPay includes maximizing the successful employee direct deposit enrollments prior to their groups launch. For Group 2, HawaiiPay decided to institute a competition among the departments to encourage departments to motivate their employees toward early direct deposit enrollments by awarding departments with the highest enrollment participation a special reward. Group 2 enrollment numbers exceeded the projects expectations with six departments reaching 100% participation and a 90% overall participation rate at go-live.

Help Desk

HawaiiPay ability to support the needs of the business not only requires a software solution but also effective help desk support. Help desk capabilities of the HIP Service Center have ramped up significantly since our initial assessment with hiring of additional help desk staff and the implementation of Service Cloud along with TalkDesk integration. Help desk processes continue to be refined on a week to week basis and metrics are



Findings & REcommendations Page 17

beginning to be leveraged to improve and maximize specific Service Center capabilities. Additionally, the help desk “top five” questions are reviewed by the project on a weekly basis to provide visibility and insights into customer experience which then triggers discussions on how the project can best address issues they are seeing and to best meet their needs.

4 FINDINGS & RECOMMENDATIONS

The following are some of the key findings and recommendations relevant to deployment for Group 2. While the project has made progress in mitigating many of these risks and issues, IV&V continues to monitor to assure further risks are not realized.

# Category Finding Recommendation Rating

1 Communications Management

Undefined communication metrics and performance targets

• Re-execute Stakeholder Analysis activities to ensure all stakeholder groups’ communications needs are known, accurate, and updated • Elaborate and document how and when each stakeholder group will be addressed by the Awareness Campaigns • Define the communication metrics that should be captured for each stakeholder group to ensure they are ready to execute their tasks and transition in accordance with the project’s schedule • Define the communication performance targets for external stakeholders, and/or success criteria for each stakeholder group, so that informed implementation decisions are made based on the state of readiness of external stakeholders

Low

3 Cost and Schedule Management

Project schedules not integrated

• Though current schedule management processes appear to be effective, IV&V recommends SOH consolidate scheduled activities into a single, integrated schedule (including detailed organizational change, communication, cutover, and readiness assessment activities for stakeholders, interfaces, and Group) and incorporate CherryRoad’s milestones in order to indicate dependencies and more easily identify resource over-allocations

Medium

4 Cost and Schedule Management

Concurrent execution and production support activities for Group Implementations

• Update the schedules for Group 2 and Group 3 with tasks and lessons identified from the Group 1 pilot implementation

Low



Findings & REcommendations Page 18

• Finalize new baseline schedules for Groups 2 and 3 which confirm that all the tasks and deliverables are achievable in prescribed timeframes • Continually monitor changes to the schedule and the impact on defined implementation dates

6 Human Resource Management

Insufficient project resources

• Evaluate which project resources are needed to allow for dedicated strategic leadership in key positions (e.g. OCM and Training) and to alleviate existing project resources with multiple project leadership responsibilities. • Assign a single, dedicated strategic management lead for key areas such as OCM and Training. • Create and utilize a resource management plan to assure planful, instead of reactive, addition and management of resources. Plan should address movement of resources as project transitions to different phases (e.g. moving from DD&I to M&O). • Formalize and document (e.g. org charts, POC lists/directories) all leadership roles and project points of contact for key areas and ensure stakeholders have easy access to comprehensive project role lists that include contact info.

Medium

12 Organizational Change Management

Less than optimal OCM management structure

• Clearly define how the change agents (Super SMEs) will accomplish the following: o Complete training to ensure they understand the role o Ensure their time is sufficiently allocated to perform the Change Agent / Super SME tasks o Report to both project leadership and department leadership any issues concerns • Update the project’s roles and responsibilities (document) to clearly define the assigned resources for each OCM task • Appoint an OCM strategy manager whose primary responsibility is to own/drive the OCM strategy and help direct OCM activities • Stepped up OCM efforts to ensure the project scope and approach is clearly and often communicated • Follow through to validate communications are effectual and the message is being received by appropriate stakeholders. • Targeted communication to stakeholders who have expressed frustration and to large organizations who may have internal communication challenges. Over-communicate important messages and provide simplified, clarifying details/instructions, especially for stakeholders who may misunderstand or misconstrue messages/instructions.

Medium



Appendix A: IV&V Findings and Ratings Defined Page 19

15 Project Organization & Management

Impact of Legislative Actions

• Establish increased communication with lawmakers and legislative analysts to ensure informed legislative decisions. • Closely track legislative actions and legal cases that could impact HawaiiPay and be proactive in preparation for them.

Medium

24 Project Organization & Management

Project instituted incentives to increase department engagement

<Positive Finding> N/A

25 Quality Management

Insufficient data validation, checks and balances

* Revisit existing data validation processes and procedures (automated and otherwise) to identify which should be implemented/enhanced and prioritized based on criticality and impact to payroll processing and stakeholder confidence. Once identified, an implementation plan can be created and implemented based on available resources to mitigate this risk. * Automated data validation support can not only increase data accuracy but also reduce the level of effort of manual processes for already constrained project resources. * Explore the feasibility of having the agencies and TPA's to validate the final payroll run data before payroll is run.

Medium

26 Project Management

Lack of formal processes

The project has implemented control processes that required significant human intervention. IV&V has noted that these processes may be effective, they may also add unnecessary risk.

Medium

27 System Change Management

Lack of comprehensive regression testing

Add a release management process that includes grouping required system changes into release and execute a regression test for the entire group

Medium

28 Testing Lack of testing results Capture the results of each significant testing step and store them with the executed test scripts for further validation

Medium

29 Defect Management

Lack of single repository for all system defects

Capture all system defects in one system. This will allow for improved release management planning and over performance reporting

Medium

APPENDIX A: IV&V FINDINGS AND RATINGS DEFINED

IV&V attends meetings, reviews documentation, conducts interviews, and performs independent analysis in order to verify and validate project activities and progress. PCG defines a “finding” as a statement of observation that relates to the project. A finding may be classified as positive, preliminary concern, risk or issue.




• A positive finding presents a statement based on a fact that supports the project. Typically, these are raised to acknowledge adherence to standards and project guidelines that are identified as part of an assessment or evaluation. For example, a project performs additional testing (outside of testing requirements) to the benefit of the project.

• A preliminary concern is an item believed may pose risk to the project, but more analysis and a better understanding of the subject area is necessary before classifying the item as a formal risk or issue. Preliminary concerns are documented in statements which articulate the concern and indicate further analysis and/or understanding of the matter is required.

• A risk is an uncertain event or condition that, if it occurs, has a positive or negative effect on a project’s objectives. PCG identifies risks with negative effects and expands the definition to include both conditions which may occur and those which may not occur (e.g. lack of a well-defined requirements traceability process could lead to delivery of an incomplete system, requiring costly and time-consuming rework).

• An issue is an event, often previously identified as a risk, which has occurred and caused negative impact to the project. Issues are documented as findings which identify the event, its impact to the project, and status towards resolution.

A key to risk management is having an understanding of all the potential risks to the project and ensuring that these risks and risk mitigation strategies are communicated to key project stakeholders on an ongoing basis. Risk analysis should begin early during project planning by determining or identifying the factors that may affect the project. Risk can impact a project in many different ways: project quality, manageability, cost, and schedule. Proper risk identification seeks to determine how the risk may affect the project and to document the project area(s) impacted by the identified risk.

Once risks are identified and characterized, both qualitative and quantitative factors are examined. Our analysis examines project conditions to determine the probability of the risk being realized and the impact to the project, if the risk is realized.

The overall risk exposure rating, or priority, is derived using the Risk Rating Matrix shown in Table 3 by finding the intersection of the probability of occurrence and the magnitude of impact on the HawaiiPay Project. The exposure rating determines the priority of each risk based on an assessment of probability of occurrence and magnitude of impact. Note that Eclipse IV&V™ incorporates “Time Horizon” (short, medium, long) into the probability score such that the more time that exists to address the risk, the lower the probability of occurrence will be.

Table 2: Risk Rating Matrix




The following Table 4 defines the Risk Priorities that PCG uses when identifying risks.

Table 3: Risk Rating Definitions

Risk Priority Definition

High

Possibility of substantial impact to product quality, manageability, cost, or schedule. A major disruption is likely and the consequences would be unacceptable. A different approach is required. Mitigation strategies should be evaluated and acted upon immediately.

Medium

Possibility of moderate impact to product quality, manageability, cost, or schedule. Some disruption is likely and a different approach may be required. Mitigation strategies should be implemented as soon as feasible.

Low

Possibility of slight impact to product quality, manageability, cost, or schedule. Minimal disruption is likely and some oversight is needed to ensure that the risk remains low. Mitigation strategies should be considered for implementation when possible.

Issue Priority is determined by its impact on the Project. PCG uses the priority levels shown in Table 5 for issues:

Table 4: Issue Rating Definitions

1 2 3 4 5

Negligible Minor Moderate Significant Critical

5

Probable

(80% – 99%)

4

Likely

(60% – 79%)

3

Possible

(40% – 59%)

2

Unlikely

(20% – 39%)

1

Improbable

(1% – 19%)

Magnitude of Impact

Probability of

Occurrence

Medium Risk

High Risk

Low Risk




Issue Priority Definition

High

The issue presents substantial impact to product quality, manageability, cost, or schedule. A catastrophic disruption is likely and the consequences would be unacceptable. A different approach is required. Mitigation strategies should be evaluated and acted upon immediately.

Medium

The issue presents moderate impact to product quality, manageability, cost, or schedule. Some disruption is likely and a different approach may be required. Mitigation strategies should be implemented as soon as feasible.

Low

The issue presents slight impact to product quality, manageability, cost, or schedule. Minimal disruption is likely and some oversight is needed to ensure that the risk remains low. Mitigation strategies should be considered for implementation when possible.



Appendix B: Assessment Category Definitions and Ratings

Page 23

APPENDIX B: ASSESSMENT CATEGORY DEFINITIONS AND RATINGS

Table 6 below lists and defines the HawaiiPay Project’s assessment categories that are used throughout the project to group IV&V findings. It should be noted that, at times, findings may span more than one category.

Table 5: Assessment Category Definitions

Category Category Description *

Communications Management

Communications management is the systematic planning, implementing, monitoring, and revision of all the channels of communication within an organization, and between organizations; it also includes the organization and dissemination of new communication directives connected with an organization, network, or communications technology. Tasks defined in the communications management plan aim to gather the project information, distribute it to the stakeholders in a timely manner, and, finally, store it. This category focused on internal project communications.

Contract Management

Contract management is the oversight and management of contracts made with customers, vendors, partners, or employees. Tasks defined in contract management are aimed at ensuring compliance with the terms and conditions, as well as documenting and agreeing on any changes or amendments that may arise during its implementation or execution.

Cost and Schedule Management

Delivering a project within the time frame promised (schedule) and within the allocated budget (cost) are fundamental objectives for all projects. Schedules and budgets are interlocked, and most likely an increase in one causes an increase in the other. Tasked defined in cost management are aimed at estimating costs for changes, monitoring contract performance, and processing approvals and invoicing for contract deliverables. Tasked defined in scheduled management are aimed at estimating and sequencing work effort, establishing a schedule baseline, managing project resources’ assignments and the completion of work effort, and monitoring schedule performance.

Human Resources Management

Human resource management (HRM, or simply HR) is a function in projects designed to maximize team member performance in service of the project’s strategic objectives. Tasks defined in HRM are aimed at recruiting, training, developing, and monitoring project team members as well as managing their productivity, transition within the organization, knowledge transfer activities, and appropriate utilization.




Page 24


Knowledge Transfer

Knowledge transfer is the practical task of transferring knowledge from one part of the organization to another. Tasks associated with knowledge transfer aim to organize, create, capture or distribute knowledge and ensure its availability for future users. Knowledge transfer includes formal and informal training, project document, and online tools which convey information need to support the implementation or operations of the new system.

Operational Preparedness

Operations management is an area of management concerned with designing and controlling the process of production and redesigning business operations in the production of goods or services. It involves the responsibility of ensuring that business operations are efficient in terms of using as few resources as needed and effective in terms of meeting customer requirements. Tasks defined for operational preparedness are aimed at establishing and confirming the readiness of the technologies, organization, and end users to stand up a new system and transition to the new operations.

Organizational Change Management

Change management is a collective term for all approaches to prepare and support individuals, teams, and organizations in making organizational change. It includes methods that redirect or redefine the use of resources, business process, budget allocations, or other modes of operation that significantly change a company or organization. Organizational change management (OCM) considers the full organization and what needs to change. Tasks defined for OCM are aimed at guiding internal and external end users to adopt the new system as seamlessly as possible. This category focuses mostly on external project team communications.

Project Organization and Management

Project management is the discipline of initiating, planning, executing, controlling, and closing the work of a project team to achieve specific goals and meet specific success criteria. The project organization is the hierarchical and/or matrixed structure created to the execute the project work. Since each project is unique, project organizations and management approaches are often customized to align with current organizational procedures, capabilities, or objectives.




Page 25


Quality Management

Quality management ensures that an organization, product or service is consistent, meets project requirements and objectives, and is fit for purpose. Quality management tasks aim to plan for quality assurances and controls throughout the life of the project for not only the product or service but also the processes used to achieve it. Quality controls, or metrics, provide insight into the project’s progress and highlight areas of concern that can be improved or mitigated.

Requirements Management

Requirements management is the process of documenting, analyzing, tracing, prioritizing and agreeing on requirements and then controlling change and communicating to relevant stakeholders. It is a continuous process throughout a project. Requirements management tasks are aimed at tracking and validating requirements through the project’s life cycle to ensure the right system is being built.

Risk Management

Risk management is the identification, evaluation, and prioritization of risks followed by coordinated and economical application of resources to minimize, monitor, and control the probability or impact of unfortunate events or to maximize the realization of opportunities. Risk management tasks include identification, rating, tracking, and monitoring of both project risks and issues. Tasks also included detailed impact analysis of project risks and issues so that strategies are developed and executed to manage threats to the project.

Systems Architecture and Design

Systems Architecture links business processes to their solutions and defines how the infrastructure, applications, interfaces, batch / online processing, data flows between systems, diverse configurations, operational governance and service delivery will be integrated and managed. The architecture is used to proactively guide development and project efforts and includes: middleware, system environments, data centers, security, and network design. System architecture and design tasks include those efforts associated with building, documenting, and deploying a software solution that meets the needs of the organization and complies with organization’s technology standards and policies.

* Some Category Descriptions were derived from https://en.wikipedia.org/ and tailored for HawaiiPay.

Individual risks and issues are rated based upon qualitative and quantitative measures defined in the IV&V plan and shown in Appendix A: IV&V Findings and Ratings Defined. Category ratings distil the status of key project areas into a simple rating, with specific and prioritized recommendations for improvement. Each category will be rated based upon the overall category’s risk to project success: high, medium, and low.

https://en.wikipedia.org/




Page 26

Table 6: Assessment Category Rating Matrix

Rating Definition

High

A category rated high (also colored red), poses significant risk to project success. This category will either have an overwhelming quantity of medium and/or high risks and/or issues, or may have a specific risk or issue that presents catastrophic risk to project quality and overall success. Categories that are rated high should be given priority and will identify the major targets that caused the category to be rated as such.

Medium

A category rated medium (also colored yellow), poses moderate risk to project success and generally has some products or processes that are deficient in quality. A category rated medium will either have a preponderance of risks and/or issues or may have specific risks or issues which present substantial risk to project quality and overall success. Categories that are rated medium will identify the major risks and issues that caused the category to be rated as such.

Low

A category rated low (also colored green), poses the least risk to project success and can generally be considered to be delivering high quality products and processes. A category rated low will also generally not have significant quantities of risks or issues...

Date post:	08-Oct-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

State of Hawaii Department of Accounting and General ......HawaiiPay Project Deployment Audit Report...

Documents