13
Test as We Fight Dennis O’Donoghue Vice President, Boeing Test & Evaluation NDIA T&E Conference March 13, 2012
Test as We Fight Dennis O’Donoghue Vice President, Boeing Test & Evaluation NDIA T&E Conference March 13, 2012
Engineering, Operations & Technology | Boeing Test & Evaluation
Patterns of Complexity
N=4 L=6 P=64
Define Requirements
Approve EWA
Fund EWA
Execute EWA
Documented Process
Define Requirements
Approve EWA
Fund EWA
Execute EWA
Emergent Pattern Dominant Process
Engineering, Operations & Technology | Boeing Test & Evaluation
Nodes Links Possible Patterns
Patterns of Complexity
L=N(N-1)/2 P=2^L
N=4 L=6 P=64
N=10 L=45 P=3.5 Trillion
N=12 L=66 P=4,700 Quadrillion
Engineering, Operations & Technology | Boeing Test & Evaluation
Flight Test Operations Network
EWA Approver
TestDocumenter
Test Sequencer
Tester
FlightPlanner
DetailedTest Planner
Dispatcher
ConfigurationRequirements
Collector/Sorter
List of ProposedWork (with
identified internalpriorities)
TIS (2)
MxRequirements Collector/
Sorter
WorkPackage
Consolidator/Prioritizer
Instrumentation
Calibrator
Weight andBalance
Requirements
Collector
Instrumentation
Requirements Collector/
Sorter
List of CalibratedParts Complete,Cal Sched, datelist, Sig CND &
Xducer Cal
List of proposedwork (with priority)0
List of ProposedWork (Inst)
List ofproposedwork (with
priority)
WorkPackageSender 1
Work PackageCapacity
Reviewer/Validator
ConfigurationGatekeeper
Consolidated list of proposed workpackage with relative prioritiesidentified (includes required &
optional work)Capacity ApprovedWork Package &
Schedule
Approved Configuration - Workpackage meets program
requirements for flight dataintegrity
ScheduleMx PlanSupplier
A/P Receiver 0
WorkAssigner /Receiver
WorkPerformer 1
WorkValidator 0
ProblemIdentifier 0
AirplaneProblemResolver
FlightRelease
Signers 0
Solution
ProblemDescription
CommunicateIssues
Verify CompleteWork Package
Assign work
Check Work
Status 0
A/P Time ofReturn 0
Test Conduct: Responsible for planning, defining,authorizing, and performing all testing on the airplane.
Airplane Transition: Responsible for planning, defining, andauthorizing all daily non-test work to be done on the test
airplane, other than testing.
Maintenance, Configuration & Preflight: Responsible for perform work on the test airplane, including supporting testing when req
FP3 DesignatedProblem
Interfacer, Ifproblem affectswork package
content orschedule
Feed back
StatusVisualizer 0
Feed back 0
ProblemDescription
Executable WorkPackage 1
List of schedule info andall requirements for paper,
parts, and tools
Validated list of theavailability/ status of all
required paper, parts, andtools
Request Status(paper, parts &
tools) or EstimateInfo
FP2Designated
ProblemInterfacer
Request for more labor ifwork package exceeds
current capacity
Feedback (If yes,continue, if no, request
change)
"Outside"Other work
sourcesupplier
(mayincludeTOC)
ADs, etc.
Red LabelUpdates, etc. 2
OperationalCommunicator
FlightScheduler
FlightResourceAssigner
TIP (1)
Schedules
TIRL
RIP
TIP
TIS
Modules/Blocks
Sequence
Sequence(0)
TIS (0)
CrewAssignment
Flight Info
OperationalInfo
Test Status
TIS (1)Airplane Status,
Global Summary,Diary, Hotline, Data,
Reports
Airplane Status &Squawks (log book)
Schedules(0)
Schedule(1)
Schedules(3)
ContingencyPlanning (2)
TIPFeedback
Sequence Feedback,Conditions, Test
Participants
SequenceFeedback
ContingencyPlanning (3)
ContingencyPlanning (4)
Airplane Status,Global Summary,
Diary, Hotline,Data, Reports
Feedback
Test StatusFeedback
A/PSelection
TIPFeedback(0)
Test Scheduler
Status (1)
Status (2)
Airplane
Staller
FC Releaser
Released FC
Notice ofPlanningAvailable
Approved IPs,SOIs, NCOs
FTE Diary/ FTEASummary / PL&D /
Data
Test ConditionCompletion, Flight
Ground Hours (test log),Flight Log
Flight InfoFeedback
SchedulingQuestions
Details ofissues/problem
Operational InfoFeedback
Regulator(FAA/EASA/B
DCO )
Test Plan,Issue,
Papers, SAC TIA, SAC,Feedback
Cert Report /Data
ApprovedReport / Data
WeatherForecaster
WeatherRequirements
WeatherRequirements 1
Forecast 1
Forecast 2
Problem Resolution Owner
MxProblem
ResolutionOwner
Problem details(assignment of
problem or issue)
Controller
ServiceRequest
ClarifyingQuestions(checklist)
DesignatedProblem
Interfacer -From FPTeams
FleetOptimizati
onAnalyzer
TechnicalSupport
Problem Details &Action Items
ProblemDiscussion 1
Request MoreDetails
Authorization toExecute Solution
ClarifyingAnswers
Continual Status (untilresolution is selected and
implemented)
Details
Data Places
Problem 0
Solution
Request MoreDetails 1
Details 0
ProblemSolution Options
ProblemDiscussion
AirplaneProgramPlanner
Value Network Map - This is a WORK in PROGRESS.
Purpose: Our purpose is to continually generate optimal aircraft test excellence and efficiency through a trusted visible network thatdelivers integrated fleet testing, updates virtual real-time status, provides a single portal for problem resolution through the TOC, and
facilitates interactive communications and collaboration.*
FleetReadiness
Coordinator
DataVisualizer
Program TestPlans &
Schedules
Data
Test ReadinessIssues
PotentialImpacts
SolutionOptions (0)
Data UpdateAnalysis
SolutionStatus
A/P release andlogbook
EngrProblem
ResolutionOwner
SolutionOptions
Schedule 1CrewAvailability
Released EWA,Test Desc.
ReleasedFC 0
ALPS Liason
FOT&VALPS
Inputter
CertComplianceNegotiatorProblem
Owner(Pgm)
ProblemDefinition
ResolutionPlan
Delta to CertPlan
Non-Compliance
Daily ALPSReport
RequestAdditionalTesting Test
Questions
Answers
Agreement withBackup Plan
Decision
Test ResultsProblem
Interfacer
TestSupportPlanner
EWA Author
TOL/ ADVReviewer /Approver
CriticalPath
Manager
Financer
EWA Reviewer
Program Interface: Responsible for interfacing with the Proga define, manage and authorize the work statement to be perfor
Test GrowthRqmt
ReleasedTOL
Draft EWA
Oppor &Risk
Special TestHardware/Facilities
Rqmts
ReleasedTOL (0)
ReleasedTOL (1)
Charge Line
Request for CustApproval
Cert PlanAuthor
Cert PlanReviewer /
Focal
FT SpecAuthor
DRO Logger
ConformityRequirements
Approver
MOCQuestion
Cert Plan
MOCAnswers
TestApproval
ConformityRqmts
Approved CertPlan 0
FT Spec
ConformityRequirements
DRO Log#
Cert Rqmts
Prelim EWAInfo
ConformityRequirements &
Approval
TestSchedule
Priorities
Test PartsSparesOrder
Approver
PartsRequester
(ME)
Parts Orderer
Supplier (inclPart Maker)
Parts On DockDates
Released PartsRequest
Request forParts
ScheduleInfoPO
Parts
Clean EWA
Clarification
Concerns,Scope,
Justification
R/B Label PRR LTPRStandby Stock Req.
IDAS Input
EquipmentMgr
Contract Writer
Auth. to ChargeProgram
Contract
Test ProgramRestrictions
MaintenancePlan Rqmt
ProgramSchedule
Rqmts
EWA Info /Test Input
CriticalChain
Information
Prioritization
SchedQuestions
Revised/ReleasedEWA
TCR
FP2
FP3
FP1
FP0
WorkInstruction
Provider
.
ResourceReservation
Maker
ReleaseNotification
ServiceEngr SBReleaser
SB Release
AD/TaskCards/SB
NCR/NCOWorksheet & Other
FTWS
Worksheet
©©
©
©
©
©
©
©
©
©
©
Work pkg as definedcannot be completed (time
allotted or kit conflict)
Rev 20071130
AnalysisTeam
Leader
Operations TeamLeader
ProgramStructures
TeamLeader
ProgramSystems
TeamLeader
InstrumentationTeam
Leader
AnalysisSMEs
Operations SMEs
InstrumentationSMEs
SystemsSMEs
Structures SMEs
5thFloor
InstrumentationDesigner (May
be anothernetwork)
Test ProgramRequirements
LoggedTIA
ApprovedTIA
Released EWA,Test Desc. (1)
EWA Information/ Test Input
Prelim EWAInformation / Tech
Support
ApprovedTIA 0
Ext External to FOTV
Work PkgDefers after
Normal Cycle
Request TestChange
Test ChangeResponse
DiscreteWork ItemEstimator
RequirementsCoordinator
Coord for Tool /Equip Reservation
Coord for Tool /Equip Reservation 0
Coord for Tool /Equip Reservation 1
FP0Designated
ProblemInterfacer
Problem Details forTOC to engage Global
Partners
Flight Plan
Flight Plan 0
FAA FlightController
Part Rqmts
Part NeedRequest
WorkPackageScheduleAnalyzer
CoordinatedSchedule
InfoCapacityInfo
Balast & FuelRqmts
©
Part &ComponentTransporter
ResolutionInfo
Unresolvable Kit /Resource Conflict
SafetyCoordinat
or
CDC / AOG / Factory Coordin
ator
Kit Receiver 0
All elementsrequired to
accomplish work
People / Resource toaccomplish the work
Coordinatealternatives for Kit
Issues
Alternatives Identifiedto address kit issues
Part AvailabilityStatus
Logged TIA
Request forStatus
Part Availabililty /Schedule
Information 0
Work ItemRequirements
Work ItemRequirements
Status (parts, plans &tools) or Estimate info
Status of Parts,plans, tools or
Estimate
Request SpecialWork Need Info
Special WorkNeeds Identified
UndocumentedRequirements
Information
Flight LogData
Enterer
Flight LogInformation
Kit & PartStager
Required WorkComponents
Missing KitComponent
(Reservation)
Data Place (IP,SOI, FC, R/L
Rqmts, FTWS,NCO, Task
Cards)
CrewPlan
QA SME
MX SME
Data Sys / InstrSqawks
SB Release
ProgramEngineering
Test RequirementData
TIS
TIS Request forEngr Support
S/B Release
TIRL
Schedule
Schedule
Operational Info -Airplane Landing
Communicationwhile remote (ETA)
Early look at squawks(airplane statussquawks, etc) 0
TestStatus
Feedback0
Test StatusFeedback 0 0
Test StatusFeedback 0 0 0
Communication of problems gettingwork package completed (includingP/Us, SQUAWK, etc.) Details of any
potential Delta to Work Package
Communication of problems gettingwork package completed (includingA/P PU's, SQIAWK, etc.) Details ofany potential delta to work package
Plans toaddresschanges
Plans toaddresschanges
Plans to addresschanges
Work Packages Changes/ Deltas after Initial
Approved WP
Work Packages Changes/ Deltas (after Initial
Approved WP)
Work Packages Changes/ Deltas (after Initial
Approved WP)
Shift Manager
Decision /Direction
Chief TestPilot
Issue Info for Resolution &Communication of changes
to a/p work package
Issue Info forResolution
Visibility
Visibility
FC Author
Perlim EWAInformation
TOL / ADVAuthor
Prelim EWAInformation (2)
ReleasedEWA (1)
ReleasedEWA (2)
DraftFC
Draft TOL
Test PartsRequirement
A/PCommunications
(ETA, etc.)
FP2.5
Parts and Tools: Responsible for ensuring the availability anddelivery of parts, tools and resources to support completion of the
identified test work package .
Part / Kit AvailabililtyStatus / Schedule
Information 3
HardBallast
Provider/Initiator 0
RVI/NDIProvider 0
Sealer /Composite
RepairProvider 0
MRSAProvider/Ini
tiator 0
Tool RoomCribs Tool
Provider/Initiator 0
Red LabelAdministrat
or 0
Kit / PartsProvider/Initiator (EFP,FTS, PCA)
GSEProvider 0
SpecialTools
Provider 0
VendorReps
Parts/PaperProvider/Ini
tiator 0
Part Requirement/ Kit Request
ShippingProvider 0
ExportControl
Coordinator0
ECCN #s 0
ECCN #'sRequest 0
ECCN Paper/Parts 0
MRSA PartsRequest 0
SupplyChain
Analyzer 0
PartsProvider
(Stores/SWControl
Library 0
PartRequest 0
Part Demand /Status
PurchaseOrder
Parts Reqst0
Vendor 0
RequestAssistance 0
Assistance 0
Tool Holder 0
ToolsRequest 0 Tools 0
PartsRequest
Coord PartPickup
Coord PartPickup
PartsDelivered
PartsDelivered
Parts & Equipmentto be Transported
FieldCoordinator
0
MaintRequest
MaintainedGSE
ToolRequest 1
Tools 1
Test Operations Center: Responsible for fleet-level test program managemvisibility and metrics and managing non-local problem resolution.
ReleasedEWA
©
Released FC
ScheduleChanges
Request PPTStatus or Estiimate
Status of Fueland Ballast
Summaries of PlannedWork Package (for next
day+)
PPT Status
Daily TestStatus
StatusRequest
Status Info
Part Demand /Status
Part Status
Kit / Equipment /Part Requirements
TestSchedule
DST
A/P Test DataSystem Status
A/P ReleaseStatus
A/P in Stall
A/P in Stall
MaintenanceBriefing to Pilots
Airplane
A/P Status(ETA, etc.)
WeightManifest
Status
Parts on DockDates
ReleasedEngineering
ITPR
©
Status /Info
RequirementInfo
Availabilityinfo
RequirementInfo Availability
info
RequirementInfo
Availabilityinfo
RequirementInfo Requirement
Info
Availabilityinfo
Availabilityinfo
RequirementInfo
Availabilityinfo
Tools to beTransported / Kitted
Ballast to beTransported GSE to be
Transported
TIS
Work NeedsDelivered to Work
Location
Pull Data forCharts
ElevatedIssue
ReleasedTOL
Request Status(PPT)
ReservedComponents to be
moved
ReleasedEWA
Final Cert Plan(Proposed)
Approved CertPlan
TOLRequirement
ReleasedFC
FC Enteredinto TICS
RIP Info inTIRL
Request forEstimate
Estimates
EstimateInformation
TIS (to TOC)- Next daytesting info
ScheduleCreator
PreliminarySchedule Info
ScheduleInfo
Executable WorkPackage
ScheduleInfo
Propulsion TeamLeader
Regulator
Visibility
Visibility
Solution
Notification ofSolution
Solution to beImplemented (from
TOC)
Solution to beImplemented (from
TOC)
FlightPlan
EngineeringData
FTIR
AirplaneLanding
Task Cards
Liaison
Feedback
Missing KitComponent (Part)
Emergent Kit /Resource Solution
Cert PlanRequirement
ReleasedFC 1
TCR / ConfigRqmts
Request forApproval
EmergentApproved EWA
CR
Config Info
Config Info 0
SMEConsultation
SMEConsultation 0
Solution to beImplemented
ConformityCoordinator
Updatet TCRConformityComplete
FAA Pickup
Rqmts forshop assistif required
Request forConformityInspection
ConformityInspector
InspectionResults
ConformityPackage
ConformanceStatement(8130-10)
Official ConformityRelease (8100-1)
ReleasedTOL 0
ReleasedTOL 1
QualityRecords
(QAR)Inputter
All Squawks
Verify CompleteWork Package
TIPS
SolutionOptions 0
FleetImpacts 0
FleetImpacts
Status MetricsVisibility
ChangeManagement
Process(metrics/vis)
Decision onSchedule Change
MeasurementPreflt Results
Problem /EngineeringSquawks 0
ReleasedTOL 2
Notification ofIssues
QAR Data
MRBDispositio
ner
NCR (EPD)Discovered at A/P
All Pickups
MeasurementPreflight Results
Non Inst MRBDisposition
MRBVisibility
InstrumentationMRB Disposition
Input Adams RIPDatabases
MeasurementPreflight Status
Decision to go toBackup Plan
TestSchedules
FTWS /PFTWS
PartsRequirements
ConfigurationPlanner
ReleasedEWA
Outside WorkRequirements
Supplier
Production Changes& Extended NCRs
Prelim EWAInformation
TIRL 0
TCR 0
Test LogisticsCoordination
Test LogisticsCoordination 0
TCR / TIPFeedback
TCR / TIPFeedback 0
Test LogisticsCoodination 2
Test LogisticsCoordination 1
Modules &Blocks
TestSchedules 0
ConfigStatus
Instrumentation Planner
TestSchedule 0
Inst Status
Inst Status 0
FTIRClarification
ContingencyPlanning
ContingencyPlanning 0
Contingency
PFTWS
List of Calib PartRqmts, Sig CND &
Xducer Calid
InstrumentationParts Request (IPR)
InstrumentationDesing Rqmts
Package
PFTWS 0
Detailed System /General Info Flow
Detailed System /General Info Flow 0
Task CardDWI
Non-Inst MRBTag
DWI EstimateIssues
DWI EstimateIssues 0
DWI EstimateAssistance
DWI EstimateAssistance 0
DWI & Dates
DWI &Dates 1
FTIR
WP CapacityIssues
FTTIR Part 1
DWI Data &Dates
DWI Data &Dates 0
FC
TCR 1
WeightManifest
ReleaseForms
FC 0
A/P ReadinessIssues
CompletedFT-TIR
Sequence 1
Issue Info forResolution
SequenceFeedback1
TIPQuestions
TIP Answers
+
+
+
+
+
+
+
Engineering, Operations & Technology | Boeing Test & Evaluation
BT&E Organization – The Visible Structure
Test Operations
Operational Support
Functional Excellence
Vice President Boeing Test & Evaluation Dennis O’Donoghue
Finance Rick Fichera
IT Business Partner Michael Stothers
Business Integration Bill Gardner
Vice President Flight Test Operations
Barb Cosgrove
Flight Operations Chuck Killberg
Test Support Operations Pete Lohr
Communications Sandy Angers
Human Resources
Susan Abbott
Ethics Dave
Young
Legal Amy Reinhart
Gaffney
Safety Suzanne
Riney
Vice President Laboratory Test Operations
Marshall Short
Test Programs Management
Sandra Gianotas
Quality Larry
Olguin
EO&T Chief Engineer Mark Burgess
Employee Recognition Linda Topp
Engineering, Operations & Technology | Boeing Test & Evaluation
Flight Test Operations Network
EWA Approver
TestDocumenter
Test Sequencer
Tester
FlightPlanner
DetailedTest Planner
Dispatcher
ConfigurationRequirements
Collector/Sorter
List of ProposedWork (with
identified internalpriorities)
TIS (2)
MxRequirements Collector/
Sorter
WorkPackage
Consolidator/Prioritizer
Instrumentation
Calibrator
Weight andBalance
Requirements
Collector
Instrumentation
Requirements Collector/
Sorter
List of CalibratedParts Complete,Cal Sched, datelist, Sig CND &
Xducer Cal
List of proposedwork (with priority)0
List of ProposedWork (Inst)
List ofproposedwork (with
priority)
WorkPackageSender 1
Work PackageCapacity
Reviewer/Validator
ConfigurationGatekeeper
Consolidated list of proposed workpackage with relative prioritiesidentified (includes required &
optional work)Capacity ApprovedWork Package &
Schedule
Approved Configuration - Workpackage meets program
requirements for flight dataintegrity
ScheduleMx PlanSupplier
A/P Receiver 0
WorkAssigner /Receiver
WorkPerformer 1
WorkValidator 0
ProblemIdentifier 0
AirplaneProblemResolver
FlightRelease
Signers 0
Solution
ProblemDescription
CommunicateIssues
Verify CompleteWork Package
Assign work
Check Work
Status 0
A/P Time ofReturn 0
Test Conduct: Responsible for planning, defining,authorizing, and performing all testing on the airplane.
Airplane Transition: Responsible for planning, defining, andauthorizing all daily non-test work to be done on the test
airplane, other than testing.
Maintenance, Configuration & Preflight: Responsible for perform work on the test airplane, including supporting testing when req
FP3 DesignatedProblem
Interfacer, Ifproblem affectswork package
content orschedule
Feed back
StatusVisualizer 0
Feed back 0
ProblemDescription
Executable WorkPackage 1
List of schedule info andall requirements for paper,
parts, and tools
Validated list of theavailability/ status of all
required paper, parts, andtools
Request Status(paper, parts &
tools) or EstimateInfo
FP2Designated
ProblemInterfacer
Request for more labor ifwork package exceeds
current capacity
Feedback (If yes,continue, if no, request
change)
"Outside"Other work
sourcesupplier
(mayincludeTOC)
ADs, etc.
Red LabelUpdates, etc. 2
OperationalCommunicator
FlightScheduler
FlightResourceAssigner
TIP (1)
Schedules
TIRL
RIP
TIP
TIS
Modules/Blocks
Sequence
Sequence(0)
TIS (0)
CrewAssignment
Flight Info
OperationalInfo
Test Status
TIS (1)Airplane Status,
Global Summary,Diary, Hotline, Data,
Reports
Airplane Status &Squawks (log book)
Schedules(0)
Schedule(1)
Schedules(3)
ContingencyPlanning (2)
TIPFeedback
Sequence Feedback,Conditions, Test
Participants
SequenceFeedback
ContingencyPlanning (3)
ContingencyPlanning (4)
Airplane Status,Global Summary,
Diary, Hotline,Data, Reports
Feedback
Test StatusFeedback
A/PSelection
TIPFeedback(0)
Test Scheduler
Status (1)
Status (2)
Airplane
Staller
FC Releaser
Released FC
Notice ofPlanningAvailable
Approved IPs,SOIs, NCOs
FTE Diary/ FTEASummary / PL&D /
Data
Test ConditionCompletion, Flight
Ground Hours (test log),Flight Log
Flight InfoFeedback
SchedulingQuestions
Details ofissues/problem
Operational InfoFeedback
Regulator(FAA/EASA/B
DCO )
Test Plan,Issue,
Papers, SAC TIA, SAC,Feedback
Cert Report /Data
ApprovedReport / Data
WeatherForecaster
WeatherRequirements
WeatherRequirements 1
Forecast 1
Forecast 2
Problem Resolution Owner
MxProblem
ResolutionOwner
Problem details(assignment of
problem or issue)
Controller
ServiceRequest
ClarifyingQuestions(checklist)
DesignatedProblem
Interfacer -From FPTeams
FleetOptimizati
onAnalyzer
TechnicalSupport
Problem Details &Action Items
ProblemDiscussion 1
Request MoreDetails
Authorization toExecute Solution
ClarifyingAnswers
Continual Status (untilresolution is selected and
implemented)
Details
Data Places
Problem 0
Solution
Request MoreDetails 1
Details 0
ProblemSolution Options
ProblemDiscussion
AirplaneProgramPlanner
Value Network Map - This is a WORK in PROGRESS.
Purpose: Our purpose is to continually generate optimal aircraft test excellence and efficiency through a trusted visible network thatdelivers integrated fleet testing, updates virtual real-time status, provides a single portal for problem resolution through the TOC, and
facilitates interactive communications and collaboration.*
FleetReadiness
Coordinator
DataVisualizer
Program TestPlans &
Schedules
Data
Test ReadinessIssues
PotentialImpacts
SolutionOptions (0)
Data UpdateAnalysis
SolutionStatus
A/P release andlogbook
EngrProblem
ResolutionOwner
SolutionOptions
Schedule 1CrewAvailability
Released EWA,Test Desc.
ReleasedFC 0
ALPS Liason
FOT&VALPS
Inputter
CertComplianceNegotiatorProblem
Owner(Pgm)
ProblemDefinition
ResolutionPlan
Delta to CertPlan
Non-Compliance
Daily ALPSReport
RequestAdditionalTesting Test
Questions
Answers
Agreement withBackup Plan
Decision
Test ResultsProblem
Interfacer
TestSupportPlanner
EWA Author
TOL/ ADVReviewer /Approver
CriticalPath
Manager
Financer
EWA Reviewer
Program Interface: Responsible for interfacing with the Proga define, manage and authorize the work statement to be perfor
Test GrowthRqmt
ReleasedTOL
Draft EWA
Oppor &Risk
Special TestHardware/Facilities
Rqmts
ReleasedTOL (0)
ReleasedTOL (1)
Charge Line
Request for CustApproval
Cert PlanAuthor
Cert PlanReviewer /
Focal
FT SpecAuthor
DRO Logger
ConformityRequirements
Approver
MOCQuestion
Cert Plan
MOCAnswers
TestApproval
ConformityRqmts
Approved CertPlan 0
FT Spec
ConformityRequirements
DRO Log#
Cert Rqmts
Prelim EWAInfo
ConformityRequirements &
Approval
TestSchedule
Priorities
Test PartsSparesOrder
Approver
PartsRequester
(ME)
Parts Orderer
Supplier (inclPart Maker)
Parts On DockDates
Released PartsRequest
Request forParts
ScheduleInfoPO
Parts
Clean EWA
Clarification
Concerns,Scope,
Justification
R/B Label PRR LTPRStandby Stock Req.
IDAS Input
EquipmentMgr
Contract Writer
Auth. to ChargeProgram
Contract
Test ProgramRestrictions
MaintenancePlan Rqmt
ProgramSchedule
Rqmts
EWA Info /Test Input
CriticalChain
Information
Prioritization
SchedQuestions
Revised/ReleasedEWA
TCR
FP2
FP3
FP1
FP0
WorkInstruction
Provider
.
ResourceReservation
Maker
ReleaseNotification
ServiceEngr SBReleaser
SB Release
AD/TaskCards/SB
NCR/NCOWorksheet & Other
FTWS
Worksheet
©©
©
©
©
©
©
©
©
©
©
Work pkg as definedcannot be completed (time
allotted or kit conflict)
Rev 20071130
AnalysisTeam
Leader
Operations TeamLeader
ProgramStructures
TeamLeader
ProgramSystems
TeamLeader
InstrumentationTeam
Leader
AnalysisSMEs
Operations SMEs
InstrumentationSMEs
SystemsSMEs
Structures SMEs
5thFloor
InstrumentationDesigner (May
be anothernetwork)
Test ProgramRequirements
LoggedTIA
ApprovedTIA
Released EWA,Test Desc. (1)
EWA Information/ Test Input
Prelim EWAInformation / Tech
Support
ApprovedTIA 0
Ext External to FOTV
Work PkgDefers after
Normal Cycle
Request TestChange
Test ChangeResponse
DiscreteWork ItemEstimator
RequirementsCoordinator
Coord for Tool /Equip Reservation
Coord for Tool /Equip Reservation 0
Coord for Tool /Equip Reservation 1
FP0Designated
ProblemInterfacer
Problem Details forTOC to engage Global
Partners
Flight Plan
Flight Plan 0
FAA FlightController
Part Rqmts
Part NeedRequest
WorkPackageScheduleAnalyzer
CoordinatedSchedule
InfoCapacityInfo
Balast & FuelRqmts
©
Part &ComponentTransporter
ResolutionInfo
Unresolvable Kit /Resource Conflict
SafetyCoordinat
or
CDC / AOG / Factory Coordin
ator
Kit Receiver 0
All elementsrequired to
accomplish work
People / Resource toaccomplish the work
Coordinatealternatives for Kit
Issues
Alternatives Identifiedto address kit issues
Part AvailabilityStatus
Logged TIA
Request forStatus
Part Availabililty /Schedule
Information 0
Work ItemRequirements
Work ItemRequirements
Status (parts, plans &tools) or Estimate info
Status of Parts,plans, tools or
Estimate
Request SpecialWork Need Info
Special WorkNeeds Identified
UndocumentedRequirements
Information
Flight LogData
Enterer
Flight LogInformation
Kit & PartStager
Required WorkComponents
Missing KitComponent
(Reservation)
Data Place (IP,SOI, FC, R/L
Rqmts, FTWS,NCO, Task
Cards)
CrewPlan
QA SME
MX SME
Data Sys / InstrSqawks
SB Release
ProgramEngineering
Test RequirementData
TIS
TIS Request forEngr Support
S/B Release
TIRL
Schedule
Schedule
Operational Info -Airplane Landing
Communicationwhile remote (ETA)
Early look at squawks(airplane statussquawks, etc) 0
TestStatus
Feedback0
Test StatusFeedback 0 0
Test StatusFeedback 0 0 0
Communication of problems gettingwork package completed (includingP/Us, SQUAWK, etc.) Details of any
potential Delta to Work Package
Communication of problems gettingwork package completed (includingA/P PU's, SQIAWK, etc.) Details ofany potential delta to work package
Plans toaddresschanges
Plans toaddresschanges
Plans to addresschanges
Work Packages Changes/ Deltas after Initial
Approved WP
Work Packages Changes/ Deltas (after Initial
Approved WP)
Work Packages Changes/ Deltas (after Initial
Approved WP)
Shift Manager
Decision /Direction
Chief TestPilot
Issue Info for Resolution &Communication of changes
to a/p work package
Issue Info forResolution
Visibility
Visibility
FC Author
Perlim EWAInformation
TOL / ADVAuthor
Prelim EWAInformation (2)
ReleasedEWA (1)
ReleasedEWA (2)
DraftFC
Draft TOL
Test PartsRequirement
A/PCommunications
(ETA, etc.)
FP2.5
Parts and Tools: Responsible for ensuring the availability anddelivery of parts, tools and resources to support completion of the
identified test work package .
Part / Kit AvailabililtyStatus / Schedule
Information 3
HardBallast
Provider/Initiator 0
RVI/NDIProvider 0
Sealer /Composite
RepairProvider 0
MRSAProvider/Ini
tiator 0
Tool RoomCribs Tool
Provider/Initiator 0
Red LabelAdministrat
or 0
Kit / PartsProvider/Initiator (EFP,FTS, PCA)
GSEProvider 0
SpecialTools
Provider 0
VendorReps
Parts/PaperProvider/Ini
tiator 0
Part Requirement/ Kit Request
ShippingProvider 0
ExportControl
Coordinator0
ECCN #s 0
ECCN #'sRequest 0
ECCN Paper/Parts 0
MRSA PartsRequest 0
SupplyChain
Analyzer 0
PartsProvider
(Stores/SWControl
Library 0
PartRequest 0
Part Demand /Status
PurchaseOrder
Parts Reqst0
Vendor 0
RequestAssistance 0
Assistance 0
Tool Holder 0
ToolsRequest 0 Tools 0
PartsRequest
Coord PartPickup
Coord PartPickup
PartsDelivered
PartsDelivered
Parts & Equipmentto be Transported
FieldCoordinator
0
MaintRequest
MaintainedGSE
ToolRequest 1
Tools 1
Test Operations Center: Responsible for fleet-level test program managemvisibility and metrics and managing non-local problem resolution.
ReleasedEWA
©
Released FC
ScheduleChanges
Request PPTStatus or Estiimate
Status of Fueland Ballast
Summaries of PlannedWork Package (for next
day+)
PPT Status
Daily TestStatus
StatusRequest
Status Info
Part Demand /Status
Part Status
Kit / Equipment /Part Requirements
TestSchedule
DST
A/P Test DataSystem Status
A/P ReleaseStatus
A/P in Stall
A/P in Stall
MaintenanceBriefing to Pilots
Airplane
A/P Status(ETA, etc.)
WeightManifest
Status
Parts on DockDates
ReleasedEngineering
ITPR
©
Status /Info
RequirementInfo
Availabilityinfo
RequirementInfo Availability
info
RequirementInfo
Availabilityinfo
RequirementInfo Requirement
Info
Availabilityinfo
Availabilityinfo
RequirementInfo
Availabilityinfo
Tools to beTransported / Kitted
Ballast to beTransported GSE to be
Transported
TIS
Work NeedsDelivered to Work
Location
Pull Data forCharts
ElevatedIssue
ReleasedTOL
Request Status(PPT)
ReservedComponents to be
moved
ReleasedEWA
Final Cert Plan(Proposed)
Approved CertPlan
TOLRequirement
ReleasedFC
FC Enteredinto TICS
RIP Info inTIRL
Request forEstimate
Estimates
EstimateInformation
TIS (to TOC)- Next daytesting info
ScheduleCreator
PreliminarySchedule Info
ScheduleInfo
Executable WorkPackage
ScheduleInfo
Propulsion TeamLeader
Regulator
Visibility
Visibility
Solution
Notification ofSolution
Solution to beImplemented (from
TOC)
Solution to beImplemented (from
TOC)
FlightPlan
EngineeringData
FTIR
AirplaneLanding
Task Cards
Liaison
Feedback
Missing KitComponent (Part)
Emergent Kit /Resource Solution
Cert PlanRequirement
ReleasedFC 1
TCR / ConfigRqmts
Request forApproval
EmergentApproved EWA
CR
Config Info
Config Info 0
SMEConsultation
SMEConsultation 0
Solution to beImplemented
ConformityCoordinator
Updatet TCRConformityComplete
FAA Pickup
Rqmts forshop assistif required
Request forConformityInspection
ConformityInspector
InspectionResults
ConformityPackage
ConformanceStatement(8130-10)
Official ConformityRelease (8100-1)
ReleasedTOL 0
ReleasedTOL 1
QualityRecords
(QAR)Inputter
All Squawks
Verify CompleteWork Package
TIPS
SolutionOptions 0
FleetImpacts 0
FleetImpacts
Status MetricsVisibility
ChangeManagement
Process(metrics/vis)
Decision onSchedule Change
MeasurementPreflt Results
Problem /EngineeringSquawks 0
ReleasedTOL 2
Notification ofIssues
QAR Data
MRBDispositio
ner
NCR (EPD)Discovered at A/P
All Pickups
MeasurementPreflight Results
Non Inst MRBDisposition
MRBVisibility
InstrumentationMRB Disposition
Input Adams RIPDatabases
MeasurementPreflight Status
Decision to go toBackup Plan
TestSchedules
FTWS /PFTWS
PartsRequirements
ConfigurationPlanner
ReleasedEWA
Outside WorkRequirements
Supplier
Production Changes& Extended NCRs
Prelim EWAInformation
TIRL 0
TCR 0
Test LogisticsCoordination
Test LogisticsCoordination 0
TCR / TIPFeedback
TCR / TIPFeedback 0
Test LogisticsCoodination 2
Test LogisticsCoordination 1
Modules &Blocks
TestSchedules 0
ConfigStatus
Instrumentation Planner
TestSchedule 0
Inst Status
Inst Status 0
FTIRClarification
ContingencyPlanning
ContingencyPlanning 0
Contingency
PFTWS
List of Calib PartRqmts, Sig CND &
Xducer Calid
InstrumentationParts Request (IPR)
InstrumentationDesing Rqmts
Package
PFTWS 0
Detailed System /General Info Flow
Detailed System /General Info Flow 0
Task CardDWI
Non-Inst MRBTag
DWI EstimateIssues
DWI EstimateIssues 0
DWI EstimateAssistance
DWI EstimateAssistance 0
DWI & Dates
DWI &Dates 1
FTIR
WP CapacityIssues
FTTIR Part 1
DWI Data &Dates
DWI Data &Dates 0
FC
TCR 1
WeightManifest
ReleaseForms
FC 0
A/P ReadinessIssues
CompletedFT-TIR
Sequence 1
Issue Info forResolution
SequenceFeedback1
TIPQuestions
TIP Answers
+
+
+
+
+
+
+
Engineering, Operations & Technology | Boeing Test & Evaluation
The Changing Face of Warfare in the 21st Century
“…And just like their allies in al Qaeda, this new Taliban is more network than Army, more a community of interest than corporate structure. For the U.S. military that I had spent my life in, this was not an easy insight to come by………It was over the course of years and with considerable frustration that we came to understand how the emerging networks of Islamist insurgents are fundamentally different from any enemy we have ever faced. In bitter bloody fights….it became clear to me that to defeat a networked enemy we had to become a network ourselves.” - General Stanley McChrystal “It takes a Network”; Foreign Policy, March/April 2011
Engineering, Operations & Technology | Boeing Test & Evaluation
What Does It Mean For T&E?
• The paradigm has shifted!
• Validation of the weapon system itself is no long sufficient
• Validation of the weapon system as a node in a network of weapon systems will be tantamount
• Understanding (and modeling) of the behavior of the network will be key to success in the networked battle-space of the future
• The new paradigm will require a shift in thinking, away from specification-based testing to……..
“Test As We Fight”
Engineering, Operations & Technology | Boeing Test & Evaluation
Test as We Fight
To improve test as we fight: • Create a relevant complex test environment.
– Stimulate network under test to drive emergent behavior – Understanding of the fidelity of hardware, emulation, modeling and
simulation – Pedigree of assets across the network available throughout test life
cycle
• Ensure the availability of critical assets across programs. – Labs and simulations – Integrated networks
• Keep a mission-level focus across the product test life cycle.
– Mission threads – Training testers to mission
• Change the test culture to meet complexity of the SOS test.
– Influence test-as-we-fight requirements that shape acquisition
Engineering, Operations & Technology | Boeing Test & Evaluation
Test as We Fight
Industry Networks
Contractor Facilities
Industry Partners
Non-DOD DOD
Government Networks
Low-fidelity Simulations Low-fidelity
Simulations
Other Programs’ Assets
Low-fidelity Simulations Low-fidelity
Simulations High-Fidelity Simulations
Re-hosted Operational
Software Re-hosted
Operational Software
Rehosted Operational
Software
Hardware Labs
Hardware Labs Hardware
Labs
Operational Assets
Operational Assets Operational
Assets Low-Fidelity Simulations
Engineering, Operations & Technology | Boeing Test & Evaluation
It Takes a Network
“Over time it became increasingly clear…… that our enemy was a constellation of fighters organized not by rank but on the basis of relationships and acquaintances, reputation and fame………All of this allowed for flexibility and an impressive ability to grow……The enemy did not convene promotion boards; the network is self-forming….After achieving some tactical success, a [young Iraqi] would market himself, make connections, gain followers, and suddenly a new node in the network would be created and absorbed. The network energy grew.” - General Stanley McChrystal “It takes a Network”; Foreign Policy, March/April 2011
Engineering, Operations & Technology | Boeing Test & Evaluation
The Leadership Challenge
“…As we learned to build an effective network, we also learned that leading that network—a diverse collection of organizations, personalities, and cultures—is a daunting challenge in itself. That struggle remains a vital, untold chapter of the history of a global conflict that is still underway.” - General Stanley McChrystal “It takes a Network”; Foreign Policy, March/April 2011
Copyright © 2012 Boeing. All rights reserved. | 13
