assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 115-18 November 2004
Automated Monitoringof
Process Compliance
Gary NatwickHarris Corporation
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 215-18 November 2004
Civil Programs
Government CommunicationsSystems Division
• $1.5B in Sales• > 6,500 Employees• ISO 9001:2000• SEI CMM Level 4
DoD Programs
Strategic Management andBusiness Development
Civil Programs National Programs
Homeland Security Programs Harris Technical Services Corporation
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 315-18 November 2004
•Historical Data•Best Practices
•Example Assets•Improvement
Requests
SubmitReuse
Improve
OrganizationalLearning
Programs
Integrated Compliance Approach
Program’sCompliance
Metric
ComplianceEvidence
XYZ
PROGRAM
PLAN
TailoringIPM
CMMI®
CommandMedia
Division
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 415-18 November 2004
CMMI®-SE/SW Staged Representation
Organizational Innovation and DeploymentCausal Analysis and Resolution5 Optimizing
4 QuantitativelyManaged
3 Defined
2 Managed
ContinuousProcessImprovement
QuantitativeManagement
ProcessStandardization
BasicProjectManagement
Organizational Process PerformanceQuantitative Project Management
Requirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidationOrganizational Process FocusOrganizational Process DefinitionOrganizational TrainingIntegrated Project ManagementRisk ManagementDecision Analysis and Resolution
Requirements ManagementProject PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management
1 Initial
Process AreasMaturity Level Focus
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 515-18 November 2004
CMMI® Process Area Categories
CMMI®
Support ProcessManagement
ProjectManagement Engineering
• Organizational Process Focus• Organizational Process Definition• Organizational Training• Organizational Process
Performance• Organizational Innovation andDeployment
• Project Planning• Project Monitoring and Control• Supplier Agreement Management• Integrated Project Management• Risk Management• Quantitative Project Management
• Requirements Management• Requirements Development• Technical Solution• Product Integration• Verification• Validation
• Configuration Management• Process and Product Quality
Assurance• Measurement and Analysis• Decision Analysis andResolution
• Causal Analysis andResolution
• Maturity Level 2• Maturity Level 3• Maturity Level 4• Maturity Level 5
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 615-18 November 2004
Integrated Process Manual
• Process Improvement• Training• Division Metrics
• Program Planning• Estimation• Program Monitoring and Control• Supplier Acquisition &Management
• Change Management
• Proposal Development• Requirements Analysis• System Architecting/Design• Design• Code and Unit Test• Fabrication and Assembly• Product Integration• Verification• Validation• Production• Field Support
• Requirements Management• Risk Management• Configuration and DataManagement
• Program Metrics• Decision Analysis andResolution
• Peer Review• Design Review• Quality Assurance• Integrated Logistics Support
IPM
ProgramLife-CycleProcesses
ProgramSupport
Processes
OrganizationalProcesses
ProgramManagementProcesses
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 715-18 November 2004
Integrated Process Format
OverviewA brief description of the process intent
Entry CriteriaState, Prerequisites, Criteria
Exit CriteriaState, Criteria
InputsNeeded work products, resources
OutputsResulting work products
Required ActivitiesMandatory tasks to implement the process
MeasuresProcess performance against plans
Organizational Improvement InformationMetrics, reusable work products
VerificationProcess compliance oversight
TailoringApproved tailoring, process specific
Implementation GuidanceCommon implementation descriptions
Supporting Documentation and AssetsApplicable organizational references
Program evidenceneeded todemonstrate IPMprocess compliance
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 815-18 November 2004
Required for:• every applicable IPM practice• every applicable program
• Optional for IPMcompliance (expected, butnot required).
•In formal CMMI® appraisals(e.g., SCAMPISM), these arerequired to corroboratedirect artifacts.
Process Compliance Evidence
• Direct Artifacts– Tangible outputs resulting directly
from implementation of a practice• e.g., plans, documents, products
• Indirect Artifacts– Artifacts that are a side-effect or
indicative of performing a practice• e.g., meeting minutes, reviews,
logs, reports, metrics
• Affirmations– Oral or written statements
confirming or supportingimplementation of the practice• e.g, interviews, questionnaires
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 915-18 November 2004
Process Compliance
1. Program Plans2. Program process baseline3. Program execution4. Compliance evidence5. QA verification6. Non-compliance mitigation
Integrated Process Manual
ProcessCompliance
Monitor(PCM)
ProgramStart-up
Program PhaseExecution
Tailoring
Program Appraisals
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1015-18 November 2004
Process Compliance Monitor (PCM)
PCM Integrates IPM & CMMI
CMMI
IPM
CMMI ProcessAreas
IPM Processes
CMMI Specific/Generic
Practices
IPM ProcessStatements
CMMI TypicalWork Products
IPM ExpectedArtifacts
(Direct/Indirect)
SCAMPI
SCAMPIArtifacts
(Direct/Indirect)
•Functional command media•GCSD best practices
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1115-18 November 2004
PCM Functionality
Process Compliance Monitor
System Management Project Management Appraisal Management Organizational Oversight
• Open/close project• Manage user access• Create/tailor project
process baseline• Add/modify evidence
and location forprocess compliance
• Monitor project processcompliance (IPM,CMMI*)
• Manage IPM content,traceability, versions(Process, section,statement, artifact,lifecycles, media x-ref)
• Manage CMMI content,traceability, versions(PA, Goal, Practice)
• Manage appraisaltypes, versions (QAAudits, SCAMPI*)
• Manage web pageaccess
• Manage user roles• Create/delete projects• Monitor tool usage and
performance
• Manage appraisalaccess (leaders,members)
• Create/closeappraisals
• Score processcompliance
• Approve/reject projectprocess baseline
• Monitor project processcompliance (IPM,CMMI)
• Monitor organizationalprocess compliance(IPM, CMMI)
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1215-18 November 2004
PCM Project Workflow
Create NewProcessBaseline
Tailor NewProcessBaseline
Submit NewProcessBaseline
Approve/RejectNew Process
Baseline
ProcessBaseline
Release NewProcessBaseline
Score ProcessAppraisal
Create ProcessAppraisal
Close ProcessAppraisal
ProcessAppraisal
Add/ModifyProcessEvidence
ProcessEvidence
Monitor ProcessCompliance
ProcessMonitoring
Project requestto DPG
Create Project Close/OpenProject
Add/ModifyProject Users
Startup & Planning
Execution
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1315-18 November 2004
PCM Roles
Score appraisal, add/modify evidence, monitor process complianceAppraisalMember
Monitor all projects & organization process complianceGlobal Reader
Project-specific: approve/reject new process baselineDivisionApprover
Read-only access to example projectsGeneral User
Release new process baseline, create/close appraisal, add/remove appraisalmember, score appraisal, add/modify evidence, monitor process compliance
AppraisalLeader
Add/remove appraisal leaders, release process baseline, create/closeappraisal, add/remove appraisal member, monitor process compliance
AppraisalAdministrator
Project-specific: tailor new process baseline, add/modify evidence, viewappraisal scores, monitor process compliance
ProjectMember
Project-specific: close/open project, add/modify users, create/tailor/submitnew process baseline, add/modify evidence, monitor process compliance
ProjectAdministrator
Administer all functionality of tool, access to all features of the system,create new projects, monitor tool usage/performance
SystemAdministrator
DescriptionUsers
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1415-18 November 2004
Project Administration
• Roles:– System Administrator: add/remove projects, project
administrators, division approvers, appraisal administrators– Project Administrator: add/remove project administrators, project
members
Create Project Close/OpenProject
Add/ModifyProject Users
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1515-18 November 2004
Project Administration
Edit project information;close/open project
Edit project user access
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1615-18 November 2004
Process Baseline
• Roles:– Project Administrator: create/tailor/submit new process baseline– Project Member: tailor new process baseline– Division Approver: approve/reject new process baseline– Appraisal Administrator: view new process baseline– Appraisal Leader: view new process baseline– Appraisal Member: view new process baseline– Global Reader: view new process baseline
• New Baseline• Filters – process, phase, artifact, tailored statements
Create NewProcessBaseline
Tailor NewProcessBaseline
Submit NewProcessBaseline
Approve/RejectNew Process
Baseline
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1715-18 November 2004
Recording Tailored Processes
• IPM tailoring is documented in the PCM tool during initialprogram planning– Specify the planned compliance and implementation of each IPM
statement and expected artifacts (optionally)– Tailoring codes:
– The documented tailoring is called the program’s “definedprocess”, establishing the approved baseline against whichprocess compliance audits are performed
Not applicable; specify rationaleN
Deviation; program alternative to IPM statement(s), or notimplemented
D
Tailored; description of tailoring must be specified(e.g., modifications meeting intent of IPM statement)
TAccept IPM statement as written (no changes)ADescriptionCode
Waiverapprovalrequired
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1815-18 November 2004
Tailoring IPM Processes – 1
To tailorCDR,expand IPMstatementand “Edit”
Tailoring Code (TC)Accept (default)Tailor (modified; no waiver)Deviation (waiver)Not applicable (outside scope)
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 1915-18 November 2004
Tailoring IPM Processes – 2
IPM Tailoring•Enter statement of IPMtailoring, rationale, andapproach to be followed•Reference the waiver (H-2634),if outside IPM tailoringguidelines
Tailoring Code (TC)Accept (default)Tailor (modified; no waiver)Deviation (waiver)Not applicable (outside scope)
Artifact Tailoring(Optional)
•Identify artifacts thatwill be used todemonstrate compliancewith program processes(tailored or not)
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2015-18 November 2004
PCM Filters – Baselines & Tailoring
Limit PCM displays toselected IPM processes
Filter IPM statements by an artifactexpected as evidence
(e.g., WBS, PWA, ECMD)
Constrain or expand IPMcontent for context of
tailoring
Tailored statements only
Changes to the processbaseline (deltas)
Limit display to those IPMstatements fully
completed prior toprogram transition
(i.e., no evidence needed;consensus scoring)
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2115-18 November 2004
Process Evidence
• Stakeholders:– Project Administrator – add/modify process evidence– Project Member – add/modify process evidence– Appraisal Administrator – view process evidence– Appraisal Leader – add/modify process evidence– Appraisal Member – add/modify process evidence– Global Reader – view process evidence
• Current baseline• Add/modify evidence and locations• Filters – process, phase, artifact, appraisal
Add/ModifyProcessEvidence
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2215-18 November 2004
Project Artifact Requirements
• The PCM identifies expected artifacts produced or available as aresult of following IPM practices– Many expected artifacts are explicit in the IPM process statement (e.g.,
documentation, forms); IPM tailoring or waivers needed to deviate– Other artifacts are derived from division best practices or common usage– Project alternatives are acceptable, as long as IPM intent is satisfied– Tailored artifacts can optionally be identified during program planning
• QA judges compliance with the approved project process baseline byinspecting the identified artifacts– Artifact exists and has appropriate content (e.g., required by IPM,
compliant with templates or checklists)– Coverage of associated program process statement
• Compliance is scored at the process statement level– Individual artifacts are not scored
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2315-18 November 2004
Entering Project Artifacts
Select “Add” tospecify artifacts for aprogram processstatement.
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2415-18 November 2004
PCM Filters – Evidence
Filter by IPM processes applicable tocurrent life cycle phase
Filter by expected or actual artifactsfor copy/paste across applicable
IPM processes(e.g., WBS, PWA, BOE, H-2517)
Constrain or expand IPM content forcontext of process/evidence entry
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2515-18 November 2004
Process Appraisal
• Stakeholders:– Appraisal Administrator – add/remove appraisal leaders, release process
baseline, create/close appraisal, add/remove appraisal member– Appraisal Leader – release new process baseline, create/close
appraisal, score appraisal, add/remove appraisal member– Appraisal Member – score appraisal
• Current baseline• When a new process baseline is released, then evidence and scores
roll forward• Score process statements• Filters – process, phase, artifact, statements to be re-scored, scored
statements
Release NewProcessBaseline
Score ProcessAppraisal
Create ProcessAppraisal
Close ProcessAppraisal
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2615-18 November 2004
Appraising Processes
• Process compliance and process strengths/ weaknessesare assessed in many ways:– IPM QA process audits– Development Cycle Readiness Reviews (phase transitions)– CMMI mini-assessments– Formal CMMI SCAMPI assessments– ISO 9001 audits– Independent program management audits– Risk assessments– Metrics
• Process appraisal results are used to characterize,measure, and improve process effectiveness andcompliance with standards– Action plans to address appraisal findings
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2715-18 November 2004
Process Compliance Scores
• Pending an appraisalNot ScoredNS
• To be appraised at a later date (i.e., the process has notyet been executed by the process and cannot beappraised)
Not YetNY
• Not applicable to the project (e.g., Code and Unit TestProcess is not applicable to a production-type program)
Not ApplicableNA
• Any situation not covered by the aboveNotImplemented
NI
• Direct artifact is absent or inadequate• Substantiated by indirect artifact/affirmation• One or more substantial weaknesses
PartiallyImplemented
PI
• Direct artifacts are present and appropriate• One or more substantial weaknesses
LargelyImplemented
LI
• Direct artifacts are present and appropriate• No substantial weaknesses
FullyImplemented
FI
AS
SE
SS
ME
NT
ST
AT
US
CO
LOR
SP
RO
CE
SS
CO
MP
LIA
NC
E
CO
LOR
S
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2815-18 November 2004
Score Process Appraisal
Score process statement;optionally, enter date validthru and/or update artifact.
A comment MUST BEentered if not FI
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 2915-18 November 2004
Compliance Scoring Rollup
Process statementcompliance scoreassigned by QA
Description ofnon-compliance
Process-levelComplianceScores
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 3015-18 November 2004
PCM Filters – Appraisals
Filter by IPM processes applicable tocurrent life cycle phase
Filter by expected or actual artifactsfor copy/paste across applicable
IPM processes(e.g., WBS, PWA, BOE, H-2517)
Constrain or expand IPM content forcontext of process/evidence entry
Review improvement progress overtime
Filter by statements needingevidence or pending appraisal
Filter by appraisal scores to helpestablish improvement actions
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 3115-18 November 2004
Process Monitoring
• Stakeholders:– Project Administrator – monitor process compliance– Project Member – monitor process compliance– Appraisal Administrator – monitor process compliance– Appraisal Leader – monitor process compliance– Appraisal Member – monitor process compliance– Global Reader – monitor process compliance
• Current or historical baselines• Scoring aggregation, bands, and counts• Filters – process, phase, artifact, appraisal, scored
statements
Monitor ProcessCompliance
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 3215-18 November 2004
Program Process Compliance
A
• Represents overall processcompliance score for program
• Based on lowest color score – harsh,but in keeping with CMMI® standards
B• Depicts scoring distribution over allprocess items
• More insight on the overall projectscore (A)
C• Depicts score for each process
executed or being executed by thisprogram
• 3 columns identify types ofprocesses
• In PCM, point+click on underlinedacronym drills down to scoringdetails for the process
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 3315-18 November 2004
Lessons Learned
• Establish an implementation guide for how the CMMI® isimplemented in organizational/project processes– Internal users (projects, managers, DPG/EPG)– External users (customers, appraisal teams)
• Trade-off how much projects must understand CMMI®details– Organization/project process knowledge vs. model knowledge
• Facilitate efficient on-line access and review– Process baseline tailoring– Evidence entry and appraisal– Compliance monitoring
• References to evidence must be very specific– Expected evidence– Project evidence and location (hyperlinked files/directories)
assuredcommunications™Automated Monitoring of Process ComplianceCMMI® Technology Conference & User Group 2004
Gary Natwick - 3415-18 November 2004
Gary Natwick [email protected]• SEI-Authorized CMMI® Instructor• SEI-Authorized SCAMPISM Lead Appraiser• SEI-Authorized SCAMPISM B&C Team Leader
Harris Corporation http://www.harris.com/P.O. Box 37Melbourne, Florida 32902-0037
Licensed by the Software Engineering Institute (SEI) to provide:• SEI Introduction to CMMI® courses• SEI SCAMPISM Appraisal Services
Contact Information
Capability Maturity Model Integration and CMMI are registered with the U.S. Patent and Trademark Office.SCAMPI is a service mark of Carnegie Mellon University.