18
SWAT Data-Driven IV&V Decision Support NASA IV&V Software Assurance Tools (SWAT)
SW
AT
Data-Driven IV&V Decision Support
NASA IV&V Software Assurance Tools (SWAT)
SW
ATOutline
• What is Decision Support?• Data Availability (On-Demand, Linked,
Warehouse)• Data Presentation• Analysis, Reporting and Metrics• Related IV&V Efforts• Questions
2
SW
ATWhat is Decision Support?
• For our purposes, Decision Support will be defined as helping people make decisions that are supported by relevant data
• A decision is defined as the choice of one among a number of alternatives
• Decision making refers to the overall process of making a choice, which includes assessing available data and identifying alternatives
3
SW
ATData Availability
• Data must first be made available for decision support
4
On-Demand Data Data WarehouseLinked Data
Data Availability
SW
ATOn-Demand Data
• IV&V can support daily decision making with basic on-demand data access
• Example #1 – Project PM may ask “Based on the issues we have identified so far, where should we focus IV&V resources?”ORBIT (Integrity) Queries configured to show current
project IV&V Issues by category and severity• Example #2 – Project PM may ask “Which of my team
members should I allocate to an upcoming milestone review?”JIRA Dashboard view configured to show current IV&V
project tasking status by assignee5
SW
ATOn-Demand Data
• Example #3 – IVVO Lead may ask “Which project risks currently require greater attention?”RiskManager 5x5 matrix view of risks filtered for
the IV&V Office level
6
SW
ATLinked Data
• Linked Data for our purposes is defined as structured data from multiple sources that has been related in some way which can make other useful knowledge available
7
SW
ATLinked Data
• Each data entity is given an individual identity for access, e.g. a URL can be used if accessing through the web
• The advantage comes from leveraging the relationships between data, not just the individual content itself
• Linked data concepts vs. separate “silos” of data in spreadsheets for example provides the following benefits: Data remains in an appropriate context, as it is referenced
through a common interface (e.g. web service or centralized data store )
Data is linked (related), enabling information to be combined across the organization or even from outside the organization
Data can still be accessible from other applications that depend on live data, while not preventing static data dumps where applicable (e.g. export to operational data store)
8
SW
ATLinked Data
• VocabularyDefining a formal and shared vocabulary is
necessary to document relationships (ontology)Permits “inferences” to be made when analyzing
data• Search
Some value can be gained from just traversing linked data
Greater value is provided by some search (query) capability of linked data
9
SW
ATLinked Data
10
IVVO Lead: “What types of IV&V issues or risks have we been identifying for missions with similar characteristics?”
Data linked or related by project (or mission)
SQL Server Report generated based on data that is extracted from multiple data sources
(e.g. IV&V issue tracking system, RiskManager tool, IV&V Projects/Mission
database) into a centralized operational data store
SW
ATLinked Data
11
IV&V Project PM: “Which of my team resources may be overloaded with active issue follow up and additional tasking?”
Data linked or related by analyst
Dashboard display of team assignments for active IV&V issues (ORBIT) and other tasking (JIRA)
SW
ATData Warehouse
• Historical analysis and trending must be based on data that is not just a single snapshot in time, rather data this is accumulated over a time period
• Robust and strategic IV&V decision making can be supported through a data warehouse approach that adds a time (historical) dimension to the data
12
SW
ATData Warehouse
13
SW
ATData Warehouse
14
TQ&E Lead: “Is there any correlation between IV&V issue generation rates and the maturity of methods
employed by IV&V teams?”
Data is analyzed over a period of time
Business Intelligence (BI) tool configured to aggregate the
necessary facts for analysis of any relationship
Senior Management: “Do projects with senior level analysts generate more issues/higher severity
issues?”
SW
ATData Presentation
• Data must also be presented for decision support• Data can be presented in any number of ways for
analysis:Dashboard views of data in simple table and chart
format, including custom interfaces that access other data sources to populate these views
SQL Server Reports delivered through a web interfaceExcel can be populated from outside data to leverage
additional analysis capabilityVisualized data can be presented through Business
Intelligence (BI) dashboards to help expose data patterns, trends and correlations
15
SW
ATAnalysis, Reporting and Metrics
• Online Analytical Processing (OLAP) – multidimensional data and aggregations for analysis
• Ad hoc reporting provided through such mechanisms as SQL Server Reporting Services (SSRS), with reports provided in web interface and e-mail subscriptions
• Metrics (e.g. IV&V method utilization over time by projects)
• Export to more lightweight options like Excel for specific analysis
16
SW
ATRelated IV&V Efforts
• IVVO Data Management – related FY14 initiatives identified
• Analysis Management Framework (AMF) domain model for IV&V analysis execution
• Development of dashboards for SWAT developed tools that pull from various data sources
• RiskManager and COMPASS tool web services• Atlassian gadgets for JIRA and Confluence
17
