Communication and Project Management Lesson Learned from the

© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1

K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D

A G E N C Y : H H S

P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E

C O N T R A C T ( D I T I C )

Communication and Project Management Lesson Learned from the DITIC Project

Situation

The DITIC (Data Information Technology Infrastructure Contract) and DACCC (Data Analysis

Coordination and Consolidation Center) are two separate but related contracts enabling CSAP

(Center for Substance Abuse Prevention) to meet GPRA (Government Performance Results Act) and

NOMs (National Outcome Measures) reporting requirements, and at the same time, empower State,

community, and grantee prevention organizations to implement SAMHSA’s Strategic Prevention

Framework (SPF) while improving accountability, performance and effectiveness.

This situation significantly increased the complexity of communication between key stakeholders

(DITIC and its GPO, DACCC and its GPO, as well as POs for other programs) which directly affected

the project management.

Challenge

Due to differences in roles, interests, and backgrounds, these stakeholders have different frames of

reference. They speak to each other; however, they do not always communicate. Often, the same

words have different meanings to different stakeholders. They are speaking different “languages.”

The miscommunication, if not well managed, can perpetuate throughout the IT solution design,

development, and implementation processes. We experienced the serious miscommunication or lack

of communication in the first two years of the contract. This resulted in technical redesigns, repeated

engineering work, and delays in system deployments.

Solution

In the third contract year (2009), a retreat was organized by CSAP in Boston. Most team members

from both DACCC and DITIC spent 2 days together to share each other’s work and discuss how

to improve communication and collaboration. Communication protocols and standard operation

procedures (SOP) were established between the two contractors and GPOs. After the retreat, a joint

monthly DACCC/DITIC project meeting with GPOs from both contracts was established. Project

managers from DACCC and DITIC jointly develop meeting agendas, conduct the meeting, and

prepare meeting minutes. In addition, a project management SharePoint website accessible by all

stakeholders was established to post and track project events, decisions, and progress. DACCC and

DITIC also conducted many system training together to address the needs of grantees.

L E S S O N S L E A R N E D


The improvement in communication made the project management in the last two and a half

years of the contract much more effective resulting in on-time, high-quality system development,

modification, and deployment. At the end of contract period, both contracts and GPOs felt that the

two contractors worked together like one team.

Lesson Learned

Through this experience, we have learned three valuable lessons:

1) Do not make false assumptions on quality of communication. Below are examples:

•Thecustomersknowwhattheywantsotheyshouldprovidedetailedguidanceonwhatshouldbe

done and in which way. (They usually know the general direction and express their desires but have

limited knowledge on how to make it to work. The contractor needs to listen carefully and translate

the direction and desire into concrete conceptual models and technical design.)

•Customersandotherstakeholdersunderstandthetechnicaldesignissuesandoptionsunder

discussion. (They usually have a general sense of the issues being discussed but don’t have in-depth

knowledge of the subject matter. They also interpret the issues and solutions in their own frame

of reference. The better way is to present multiple solutions in concrete forms, i.e. wireframes,

storyboards, and prototypes so they can react to and make an informed selection.)

•Thee-mails(andattachments)sent(orcc)arereadbyeveryoneonthelist(morelikely,onlythose

directly involved read them, others either did not read thoroughly.)

•Everyonereadthee-mailcommunicationandunderstandsthecontent,(realityismorelike,only

those closely connected to the matter under discussion know what you talk about but with their own

interpretation.)

2) As an IT contractor, we must be fully aware of the different “frames of reference” used by

multiple stakeholders and be able to “translate” the words and statements of other stakeholders into

technical designs and requirements to assure that communication to the engineers is accurate and

precise.

3) We need well established and enforceable SOPs and communication protocols among

stakeholders to assure continuous, in-time, and effective communication.

DUNS 834952038

NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420

CCR CAGE CODE 8A3D3




A G E N C Y : H H S



Do the Right Thing Right Lesson Learned from the DITIC Project

The Situation

The Data Information Technology Infrastructure Contract (DITIC) was a milestone effort in supporting

CSAP’s three major goals of building capacity, enhancing effectiveness, and improving accountability.

Achieving these objectives through this contract was significant to the entire field of substance abuse

prevention in several ways:

(1) DITIC served as CSAP’s coordinated prevention knowledge and information dissemination center,

(2) DITIC became the centralized data collection and management mechanism for monitoring and

managing both block grant and other discretionary grants, and

(3) DITIC was the data repository and report generation center for the field of prevention.

In conjunction with DACCC (Data Analysis Coordination and Consolidation Center), DITIC enabled

CSAP to meet GPRA (Government Performance Results Act) requirements, and at the same time,

empower State, community, and grantee prevention organizations to implement SAMHSA’s Strategic

Prevention Framework (SPF) while improving overall accountability, performance and effectiveness.

As DITIC contractor, KIT Solutions inherited 11 legacy data systems from previous vendors. These

systems were developed by multiple IT teams using different generations of technology over the past

decade. Continuing to perpetuate these fragmented systems would have hindered CSAP’s ability to

achieve the contract objectives stated above. Clearly, the right thing to do was to apply state-of-

the-art information technology to integrate these systems and create a new information technology

infrastructure.

The Challenge

After a successful transition of all the systems from the previous contractor, we received lists of

modifications and enhancement requests from the customer (CSAP) for the various inherited systems.

Due to the length of time for the contract transition, every list KIT received was a high priority, the

pressure to immediately start working on these legacy systems was great. We faced a major challenge:

modify and maintain the 11 systems to satisfy the short term program management needs of

individual systems or conduct an assessment of all legacy systems before doing any further work.



We proposed to take a pause and conduct a comprehensive assessment of all inherited IT assets

and set a vision for the project. SAMHSA’s Government Project Officer (GPO) supported this

recommendation and allowed KIT to take 2 months to conduct the assessment. The assessment

resulted in a 25-page report that established a new vision and a master schedule for the entire 5-year

contract period. The vision was to build a web portal consisting of three integrated services: (1)

Knowledge and Information Service, (2) Data Collection and Management Service, and (3) Data &

Report Service.

The Solution

Based on the recommendations from the Assessment, KIT was allowed to modify the original

DITIC scope of work using the new vision. Guided by the new vision and applied Service Oriented

Architecture (SOA), we completed the system integration by building an integrated web portal

(Prevention Management Reporting and Training System, or PMRTS) with a Single Sign On (SSO)

portal access provider. This new integrated design provides a centralized one-stop internet service

to multi-level access by CSAP administrators, project officers, program evaluators, and grantee

organizations across all program areas with a fresh and exciting interface and navigation design.

Based on the system evaluation results, the11 fragmented IT systems were either replaced or

upgraded to meet the requirements of the new technology. The contents and functionalities of the

legacy systems were re-organized under the new portal. Data collection, prevention information, and

report generation services are intelligently provided to users based on their access privileges and

grants funding streams. This integrated data portal provides data and reports that satisfy the needs of

GPRA compliance, grantee performance management, cross-site program evaluation, and ad hoc data

analysis and reporting.

Lesson Learned

While the Assessment temporarily delayed the system enhancement tasks for two months, the longer

term impact was very positive. CSAP accomplished the information systems integration within 2

years, a desired goal SAMHSA was unable to achieve in the past decade before DITIC. Through this

experience, we have learned that when you do the right thing right, temporary slow is in the long

term faster and better. As a contractor, we must think outside the box and use our best judgment to

support the mission of the customer. When a good idea is adopted by the customer, the customer

and contractor are working as partners. This type of partnership will continually lead to innovative

solutions while generating valuable Return on IT Investment.

DUNS 834952038

NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420

CCR CAGE CODE 8A3D3




A G E N C Y : H H S



Integrate Two Different Quality Improvement Frameworks Lesson Learned from the DITIC ProjectSituation

The objective of Data Information Technology Infrastructure Contract (DITIC) is to support CSAP in

meeting GPRA (Government Performance Results Act) and NOMs reporting requirements and at the

same time empower State, community, and grantee prevention organizations to implement SAMHSA’s

Strategic Prevention Framework (SPF) while improving accountability, performance and effectiveness.

This is consistent with the Strategic Initiative #7: Data, Outcomes, and Quality in SAMHSA’s 2010 –

2014 Plan (Leading Change: A Plan for SAMHSA’s Roles and Actions). The purpose of this initiative

is “Realizing an integrated data strategy and a national framework for quality improvement in

behavioral health care that will inform policy, measure program impact, and lead to improved quality

of services and outcomes for individuals, families, and communities.”

A key task of DITIC is to collect and manage data that can be used to support the implementation of

SPF and meet the national performance measures required by GPRA and NOMs. The data strategy

must be designed to improve quality of services and yield better outcomes.

Challenge

There are two different approaches to the development of a framework for quality improvement.

One is to set up standards, develop measurements, and then collect data based on measurement to

validate performance. GPRA and NOMs are examples of this approach. The idea behind this approach

is that people improve upon what is being measured. The second approach refers to technology

transfer and the implementation of high quality behavioral health services. In this approach, scientific

logic, implementation steps, and expert guidance are embedded in the data management system.

The five-step Strategic Prevention Framework (SPF) is an example of this approach. A growing body

of scientific literature suggests that intended outcomes of evidence-based practices and programs are

reliably achieved only when there is fidelity to the practice or program as it was designed and tested.

The two approaches have different implications to the type and quality of data collected, and in

turn impact future data analysis. Under the first approach (which is most common today), data

collection is simply a documentation of what happened. In the second approach, however, planning,

implementation, and evaluation data are logically connected. The fidelity of program implementation

can be assessed. To a researcher, this type of planned change data is invaluable but very hard to get.



However, to practitioners, the planned change process may be too cumbersome and impractical. The

challenge is: how to balance the two approaches in an integrated quality improvement framework?

Solution

The MRT (Management Reporting Tool) further developed under DITIC is an effort to integrate the two

different quality improvement frameworks.

With the guidance from our GPO, the KIT team expanded the use of the COMET system used for the

management, monitoring, and evaluation of the Drug-free Community Grantees to develop the MRT

as an over-arching data collection, management, and reporting platform for managing all discretionary

grant programs in support of an integrated quality improvement framework.

First, grantees from all grant programs must periodically submit NOMs and GPRA data using MRT.

An on-line, real-time report submission, review, revision, and approval service is provided by MRT for

CSAP program officers to monitor, review, and accept performance data submissions. This supports

the quality improvement framework of people improve on what is being measured.

Second, the data system for all grant programs under MRT follow the same SAMHSA SPF program

implementation model consisting of 5 steps: Assessment, Capacity, Planning, Implementation,

and Evaluation. An underlying logic model is enforced (i.e. planning must be linked to assessment

results, implementation activities must be connected to goals and objectives) electronically by the

software system. This supports the technology transfer – implementing evidence-based best practices

framework.

Finally, since both program performance data (GPRA and NOMs) and program implementation data

are stored in an integrated database, we are able to provide data extracts of performance data along

with programmatic data to the DACCC team for ad hoc data analysis as well as for accountability and

GPRA reports preparation. At the same time, we are able to provide program implementation data

to CSAP program officers and senior management for on-going grantee performance management.

Grantee organizations themselves can access both program and performance data in real-time for

continuous quality improvement of their services.

Lesson Learned

The recognition of the two different but related quality improvement frameworks was an important

revelation during our DITIC experience. Clearly, there are pros and cons for each approach. The MRT

experience taught us that it was possible to combine the two approaches to have an integrated quality

improvement framework.

We believe that SAMHSA should consider both approaches and apply them where they fit. There are

opportunities for innovation as SAMHSA establishes its national framework for quality improvement

and supports data collection platforms that allow for their integration. For example, to go beyond just

documenting what happened, we can develop intelligent data systems that guide service providers



DUNS 834952038

NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420

CCR CAGE CODE 8A3D3

to implement evidence-based best practices where they are available. This will allow us to analyze

the difference between planned change using evidence-based approaches for regular prevention

and intervention programs. At the same time, periodical data collection following established

measurements should also be consistently enforced across programs. The framework and supporting

data collection infrastructure should be flexible enough to accommodate both. The integration of both

approaches can maximize the potential for in-depth and meaningful data analysis and data driven

quality improvement in behavioral health service delivery.




A G E N C Y : H H S



Connect Grant Making and Grantee Performance ManagementLesson Learned from the DITIC ProjectSituation

DITIC (Data Information Technology Infrastructure Contract) and its related contract DACCC

(Data Analysis Consolidation and Coordination Center) were managed by The Division of System

Development (DSD) of CSAP (Center for Substance Abuse Prevention). DSD supports prevention

efforts and programs initiated and managed by both the Division of State Programs (DSP) and

Division of Community Programs (DCP) within CSAP with on-line data collection, program

management tools, data analysis, GPRA reporting, and cross-site program evaluations.

The Grant Making Process: begins with either

1. DSP or DCP as they prepare the grant solicitation RFA (Request for Application) for a specific

program;

2. DSP or DCP evaluates and awards grants to successful applicants;

3. DSD (through DITIC) designs, develops, and deploys data collection and management systems

based on the RFP and GPRA compliance requirements. and if needed, cross-site evaluation design by

the DACCC team;

4. DSD (through DITIC and DACCC) trains both grantees (data submission and reporting) and CSAP

POs (grantee performance management) to use the on-line data collection and reporting tool; and

5. DSD verifies and consolidates data, then conducts an analysis for cross-site evaluation, generates

ad hoc reports requested by CSAP management, and prepares accountability and GPRA reports for

OMB and Congress

Challenge

As the DITIC Contractor, we experienced a disconnection between program divisions (DSP and DCP)

and the system development division (DSD). DSD is responsible for standardizing the data collection

for the National Outcomes Measures (NOMs) and GPRA. DSP and DCP, on the other hand, are more

interested in the data collection needs of each specific grant program. As a result of these different

orientations, we spent a prolonged period of time revising the technical design whilst attempting to

reconcile the differences between the two departments. DSD (through DACCC) needs time (usually



6 months to a year) to design the cross-site evaluation after the RFA is issued. This led to major

system adjustments based on these new data collection requirements resulting in data collection

interruptions and retraining for grantees.

Solution

Working with the GPOs for the DITIC and DACCC contracts, the two contractors jointly developed

a new inquiry questionnaire which contained pertinent questions about the new grant RFA that

potentially affected the data collection system and evaluation design for the program (see attached

form). The form is approved by the GPOs and presented to both DSP and DCP for consideration.

While it has yet to be utilized, DSD, DSP, and DCP agree the form can fill the communication gap and

enable DSD to be engaged early on the grant making process, which in turn will speed up and smooth

out the evaluation design, implementation, and data collection system development and deployment.

Lesson Learned

Grants management has two dimensions: (1) grant making, and (2) grantee performance

management. The primary objective of grant making is to select the most promising recipients

through a fair, competitive process and assure appropriate fiscal spending. This involves RFA

preparation, solicitation and application, review and decision process, award management and

fiscal oversight. The primary objective of the grantee performance management, on the other hand,

is to monitor and assist grantees to effectively implement the grant and achieve the goals of the

grant program. This involves compliance reporting, implementation of best practices, monitoring

performance and measuring outcomes, and finally, data-driven management decisions making to

effectively allocate limited resources.

The program divisions (DSP and DCP) are involved in both grant making and grantee performance

management. However, DSD is primarily involved in supporting program divisions on grantee

performance management. This is the major cause for the disconnection between program divisions

and DSD. A conscious effort to involve DSD in the early stage of program development and RFA

preparation can help resolve this issue.

DUNS 834952038

NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420

CCR CAGE CODE 8A3D3




A G E N C Y : H H S



Balance Timeliness and Quality of Data in Supporting Prevention Program Management Lesson Learned from the DITIC ProjectSituation

As the contractor of the Data Information Technology Infrastructure Contract (DITIC), KIT Solutions

(KIT) is responsible for collecting, managing, and providing quality prevention program and outcome

data and information that enables CSAP to meet GPRA (Government Performance Results Act)

reporting requirements, while empowering State, community, and prevention grantee organizations to

implement SAMHSA’s Strategic Prevention Framework (SPF) and improve accountability, performance

and effectiveness. The users of prevention data include CSAP senior managers, Division directors,

Program Officers, Evaluators, and Grantee organizations.

Currently, grantees submit data on a biannual or annual basis. This raw data is then cleaned and

used for NOMs and GPRA performance reporting and to provide responses to ad hoc requests by

CSAP staff. The data cleaning process is quite extensive involving (1) communication with grantees

to resolve data discrepancies, (2) matching individual surveys across multiple time periods and, (3)

the identification and resolution of duplicate data entry and other data quality issues. Cleaned data

is then consolidated into program master files that reflect all data submissions to date for a given

program. While the data cleaning process served an important purpose in identifying data issues

which lead to the improvement of data quality, this process added a considerable delay to data

analysis and report generation.

Challenge

As a result of the data submission frequency (annual or biannual) and time delay (typically 5 months

after data submission) due to the data cleaning and preparation process, the performance reports are

completed a year after data submission. While these reports are meeting GPRA and NOMs reporting

requirements, they are not always focused on the needs of daily grant program administration and

grantee performance management and monitoring. CSAP senior management and program officers

need programmatic and performance data on-demand in real time or near real time for daily data

inquiries, ad-hoc reports, and decision making.

In addition, the GPRA modernization act of 2010 also requires federal agencies to use current

information technology to collect, manage, and report data more frequently (quarterly instead of

annually) and to increase the use of performance information in program decision-making.



Solution

To meet this challenge, we worked closely with our GPO and proposed a plan for developing “real-

time data reporting” (RTDR) capacity under DITIC in 2010. This plan had three key components: (1)

change grant program reporting requirements to increase grantee data submission frequency; and

(2) implement electronic data validation at data submission (front end) to assure data quality and

consistency, and (3) enable meaningful data access using web-based, on-demand report generation

and real-time data visualization.

Since the first component will take a longer period of time and involve efforts from many

stakeholders, we made significant progress in developing the second and third components.

•Wesystematicallyenhanceddatavalidationrulesforallon-linedataentrysystems.

•Wedevelopedanddeployedadatauploadingsubmissionvalidationsystem(on-line“scrubber”)

that only accepts validated data.

•Whilethedataisnoterrorfreeforanalysis,arecentdataqualitycheckbytheDACCCteam

indicated great data quality improvement and backend data cleaning has been minimized.

•WebuiltaMDM(MasterDataManagement)toolwhichcontainsperiodicallyupdateddatafrom

SAMHSA’s Grants Management Information System (SGMIS) including up to date grantee information

and project officer assignments.

•WedesignedaDataMarttohousebothsecondarydatasets(NSDUH,SYNAR,etc.),aswellas

subsets of data from our primary data collection systems (COMET, MRT, etc.)

•WedevelopedaGIS(GeographicInformationSystem)baseddatavisualizationplatform(RTDR)

providing instant data access and display for (1) grant funding, (2) program performance, (3) Targeted

Population, and (4) Prevalence Data.

•RTDRwentliveinJanuary2012andisnowprovidingreal-timedataaccessandreportingtoall

users identified above.

Lesson Learned

Timeliness and quality of Data are both critical to data driven decision support. For program

evaluation and NOMs and GPRA performance reporting, the quality of the data out-weighs the

timeliness of the data. However, for program management and grantee performance management,

timeliness is critical to the data being useful and meaningful. The table below presents the dilemma:

While the most desirable result is validated and real-time data, this is often impractical. Validated

but delayed data satisfies academic standards but has little use for on-going management decision

support. No one wants preliminary and delayed data. Preliminary (with enhanced front end data

validation) real time data can be a practical solution to address a critical need for on-going



DUNS 834952038

NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420

CCR CAGE CODE 8A3D3

management decision support. RTDR development under DITIC is an example of such a solution.


Real (or near real) Time Periodical (Delayed)

Validated Most desirable but Impractical Satisfies academic standards but has

little use for on-going management

decision support

Preliminary Practical and addresses a critical Least desirable and nobody wants it

need for management decision

support

Quality of Data Timeliness of Data

mdegruttola

Stamp

Date post:	11-Feb-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Communication and Project Management Lesson Learned from the

Documents