1. Space System Development:Lessons LearnedThe Roots of Mission AssuranceExcerptsNASA PM CHALLENGEFebruary 9, 10, 2010Joe Nieberding Used with permission

2. Introduction Excerpts from a two day presentation developed to assisttodays space system developers Explore overarching fundamental lessons derived from 35 specific mishap case histories from multiple programs Root causes not unique to times/programs Will cover some material from the two day presentation: A few of the detailed case histories Sample countermeasure principles References given for all resource information Lessons learned charts (yellow background) were either developedindependently by AEA or extracted from resource information 2 2006 All Rights Reserved. Aerospace Engineering Associates LLC 3. 35 Case Histories Covered In The Two DayPresentation Case Event CaseEventCase EventLoss of Centaur attitude controlAtlas Centaur Centaur weather shield Titan Centaur Degraded Titan Stage 2 engine Titan IVB-32 spacecraft delivered to uselessF1structural failure 6 performanceorbitAtlas Centaur Atlas booster engine shutdownAtlas Centaur Atlas thrust section in-flight On pad rain damage toGPS IIR-35 on pad vehicle destroyed 43explosionspacecraft Loss of electrical power (L+105Mars ClimateSpacecraft impacted MartianApollo 1Command Module fire Seasat days)Orbiter surfaceApollo 13 Diverging stage 2 POGO Atlas Centaur Loss of Centaur attitude controlMars Polar Uncontrolled descent to Martian POGO premature engine shutdown62due to LOX tank leak LandersurfaceApollo 13 Command Module LOX tankAtlas Centaur Loss of vehicle attitude controlX-33Program canceledExplosion explosion67following ascent lightning strikeAtlas Centaur Failure of Nose Fairing to High Gain Antenna failed toGalileo X-43A Loss of Pegasus attitude control21jettison deployAtlas Centaur Loss of control at CentaurStructural failure of spacecraft Mars ObserverLoss of contact with spacecraft CONTOUR24ignitiondue to SRM plume heating Orbiter damaged upon TOSProgram cancelled after fourLoss of control under turbulent N-1STS-51/TOS separation system Super ZipHeliosflight failures flight conditions firingAtlas CentaurInertial Reference Systems Spacecraft severely damaged inIncreased frequency of launch Launch Ariane 501 shutdown during first stageNOAA N Primeprocedure-challenged groundaborts due to upper air winds Availabilityburn vehicle destroyed handlingOrbital WorkshopSpacecraft parachutes failed to Loss of attitude control battery Skylab Micrometeoroid Shield structuralLewisGenesisdeploy some data obtained depletion; mission failurefailure after crash into the earthSpacecraft hit target; prematureTitan CentaurLoss of attitude control Centaur engines failed to start SOHODARTdepletion of attitude control1communication lost for 3 monthspropellantsAtlas Centaur Loss of Atlas control boosterLoss of primary instrument INDICATES THIS CASE INCLUDED IN THIS WIRE33separation disconnect anomalycryogens mission failureSUMMARY3 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 4. Historical Perspective: Failures Have Always Been With Us 1100 1250: Italian Master Masons learned how to build The Milano Cathedral*Gothic structures via practical skill 1250 +: Master masons moved on to other parts of Europe This skill in practice decayed as rules of practice were transferred toapprentices 1399: A wall collapsed in the Milano Cathedral Cardinal-Archbishop in Milano called in the masonsWhat would be done differently if they re-built the wall?The answer was nothing for they only knew how to build it by rote Pope convened a board of Master Masons from throughout EuropeThis Cathedral Accident Investigation Board identified the problem Lesson: Artisans engaged in operational activities, i.e.those implementing designs according to rules of thumb,lose the tactile feel for and an understanding of the basisof the rules Recovering from this loss often requires the assembly ofscarce expertise from across a field (the NASA Engineering Ars sine scientia nihil esthttp://verostko.com/mignot.htmland Safety Center is a fine example of this)*Milano Cathedral historical commentary and photograph courtesy of Dr. Joseph R. Fragola,Vice President, Valador Corporation.4 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 5. Total Number of Spacecraft Launched 1957 - 2006 Sponsor* Number% Russian3476 54% American 1735 27% European279 4% Japanese107 1.5% Chinese 103 1.5% Indian 43 0.7% Canadian 27 0.4% Israeli10 0.1% Other Government102 1.3% Commercial529 8% Amateur/Student80 1.0% Total6376* Sponsor means Spacecraft owner not always the same as the entity launching it.5 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 6. Worldwide Space Mission Success Rate by Decade 1957 - 2006 1600 1400 1200 1000Number of 800Space Missions 600 400 200 01957- 1967- 1977-1987- 1997-196619761986 19962006 Success601 1401 146212701043 Fail 232 16511289 116 Success Rate 72% 89%92% 93% 90%Decade 6 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 7. Worldwide Space Mission Failures by Mission Phase since 1957 35 End of Mission Pad: Pre-launch eventLaunch: Launch to earth Missionorbit failure 30 Orbital: Spacecraft/Upper Orbitalstage to final orbit failureMission: Mission Launchobjectives unmet 25Number of Missions PadEnd of Mission: Payloadnot recovered as intended 20 15 1050Year of Launch 7 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 8. Perspective Space system reliability has improved dramatically over a five decade history Of 6376 total space missions attempted, 694 failures occurred = 89% average rate of success First decade = 72% success Last decade = 90% success The material to follow places a magnifying glass on the relatively small fraction of space mission mishaps 8 2006 All Rights Reserved. Aerospace Engineering Associates LLC 9. A Quick Aside About Design Error Screens Design Error GIVEN: Screens Our design machine (humans)WILL produce errors at some finite rate-not zero TestDesign Error Design ReviewUnexpected Behavior Engineers today, like Galileo three and a half centuries ago, are not superhuman. They make mistakes in their assumptions, in their calculations, in their conclusions. That they make mistakes is forgivable; that they catch them is imperative. (1)(1)To Engineer is Human; Henry Petroski, Vintage Books, 19929 2006 All Rights Reserved. Aerospace Engineering Associates LLC 10. Atlas Centaur AC-5LiftoffApogee Pad Impact Category: Hardware Design/System Engineering Problem: Atlas engine shutdown twelve feet off thelauncher vehicle and pad destroyed (3/2/1965) Impact: R&D Flight; launch pad heavily damaged 10 2006 All Rights Reserved. Aerospace Engineering Associates LLC 11. Atlas Centaur AC-5 (contd) - Video 11 2006 All Rights Reserved. Aerospace Engineering Associates LLC 12. AC-5 Booster Low Pressure Fuel Duct12 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 13. Atlas Centaur AC-5 (contd) - Video 13 2006 All Rights Reserved. Aerospace Engineering Associates LLC 14. Atlas Centaur AC-5 (concluded) Why: Booster engine fuel pre-valve not fully open Flow through partially open pre-valve will completely close it Position switch design unreliable indicator of valve position LESSONS: Design to ensure fail-safe feedback to confirm proper configuration before launch commit Do not use valve designs subject to flow-induced closure Source: Subject Matter Experts: (Karl Kachigan, John Silverstein) 14 2006 All Rights Reserved. Aerospace Engineering Associates LLC 15. Galileo Category: HardwareDesign/System Engineering Problem: Partial Deployment ofhigh gain antenna (launched10/18/1989; deployment attempt4/11/1991) Impact: Significant reduction indownlink data rate Subsequent workarounds amelioratedlosses significantlySource: http://www.nasa.gov/offices/oce/llis/0492.htmlNASA Public LessonsLearned entry 0492; The Galileo High Gain Antenna Deployment Anomaly,Michael R. Johnson, JPL; NASA TM-1999-209077, Aerospace Mechanismsand Tribology Technology: Case Study15 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 16. Galileo High Gain: Antenna General Arrangement16 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 17. Galileo Failure Mode: Lubrication Breakdown Receptacle Fitting17 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 18. Galileo Failure to Fully Deploy Cause Excessive friction in the antenna mid-point restraint pin/ V - socket interface Rib preloading caused high pin socket contact stress Relative pin socket motion broke pin ceramic coating Further relative motion in air removed coating and dry lube leaving roughsurfaces Further relative motion in vacuum removed oxidized and contaminated Tileading to galling of pin and socket Asymmetric rib deployment eventually stalled ballscrew motor Contributing factor is number and configuration of cross country shippingevents Shuttle Centaur cancellation caused one additional cycle Antenna deployment testing failed to detect phenomenon Vacuum test occurred before vibration damage to pin-socket interface Ambient tests went OK due to lower coefficient of friction of Titanium pin-socket interface in air18 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 19. Galileo (concluded) LESSONS: Good design vetting is first line of defense in catching subtle design weaknesses (the V-notch was added to the TDRSS design by JPL) Needs the right kind of reviewers (multidiscipline experts) Designs that have implicit time/vibration limits need to be flagged andfactored into operational planning When unexpected and unplanned events occur Place special focus to understand if any systems impacts result Significant storage, transport and mission time was introduced into the spacecraftlife after upper stage reassignment Successful environmental testing can be dependent on some extremely subtle factors E.g. in this case only vibration testing in vacuum would have a chance ofcatching the problem 19 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 20. Atlas Centaur AC-62 Category: Production/Operations - Systems Engineering Problem: Centaur stage loss of control during coastphase (6/9/1984) Impact: Intelsat V placed in useless orbit Why: LOX tank leak Minor LOX tank leak escaped build, test and inspection procedures Small amount of SOX collected in Centaur to Interstage Adapter area SOX augmented normal shock caused by shaped charge firing Amplified shock exceeded LOX tank allowables causing four inch crack intank Escaping LOX propulsive forces not controllable in coast phaseSource: Atlas/Centaur AC-62 Failure Investigation Final Report (NASA GRC Archives) 20 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 21. Atlas Centaur AC-62 General Arrangement21 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 22. Atlas Centaur AC-62 Crack Development Sequence22 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 23. Atlas Centaur AC-62 Station 412 Structural Detail23 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 24. The Balloon Structure of Atlas (until Atlas V) and CentaurVideo A Convair innovation - design concept for the original Atlas ICBM Subsequently extended to Centaur (to the present) Propellant tanks are thin walled pressure vessels Strength and stability derived from internal pressure Advantage is very light structure (and corresponding attractive mass fraction) Disadvantage is handling complexity the need to insure tanks are always pressurized or stretched The consequence of not doing so is dramatic24 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 25. Atlas Centaur AC-62 Leak Location Photograph25 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 26. Atlas Centaur AC-62 (contd) Additional Factors: First flight with LOX tank pressure elevated ~25% (to compensate for deletion of boost pumps) Tank had been properly qualified at higher pressure However, on this tank, factory acceptance leak tests were not run at the elevated pressure for no explainable reason Original LOX tank stress analysis omitted shock effect of shaped charge firing When included however, margins still positive Leak check and X-ray procedure protocols dated State-of-the-art advances not taken advantage of Tank fabrication staff experience level significantly less than for preceding vehicles 26 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 27. Atlas Centaur AC-62 (concluded) LESSONS: Keep the test program synched up with the design Watch out for test/inspection technology insertion opportunities Analyze all relevant environments27 2006 All Rights Reserved. Aerospace Engineering Associates LLC 28. Atlas Centaur A/C-33 Booster SectionSustainer SectionAtlas Stage and a Half ConfigurationCategory: Production/Operations Program ManagementProblem: Vehicle loss of control on ascent (2/20/1975)Impact: Loss of Intelsat IV missionWhy: Atlas booster staging disconnect failed to separate Disassembly of swivel in disconnect lanyard (highly likely) Source: Atlas/Centaur A/C-33 Failure Investigation and Flight Report, Lewis Research Center, December, 1975 (NASA GRC Archives)28 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 29. Atlas Centaur A/C-33 Staging Disconnect Atlas Booster Sustainer Staging DisconnectB600P/J 1229 2006 All Rights Reserved. Aerospace Engineering Associates LLC 30. Atlas Centaur A/C-33 Staging Disconnect Lanyard 30 2006 All Rights Reserved. Aerospace Engineering Associates LLC 31. Atlas Centaur A/C-33 Staging Disconnect Lanyard Swivel 31 2006 All Rights Reserved. Aerospace Engineering Associates LLC 32. AC-33 Booster Launch Configuration and Separated 48 Inches(Ground Test) 32 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 33. Atlas Centaur A/C-33 - Observations The reliability and quality control systems were indicating swivel failures for nearly eight years, from as early as 1967! Several instances of the swivels separating into two pieces at the mating face It is incomprehensible that effective action was not taken to correct the serious problems with this system and its components The lack of follow-up and urgency suggests that the personnel involved did not understand the disastrous flight consequences that could and did occur when the system malfunctions This was truly an accident waiting to happen! 33 2006 All Rights Reserved. Aerospace Engineering Associates LLC 34. Atlas Centaur A/C-33 (concluded)LESSONS: Adopt an over-arching principle: redundancy is required inflight critical mechanisms Treat anything that performs an in-flight actuation like asystem Assign a cognizant lead engineer Ensure extra quality attention where there are unavoidablesingle points of failure (e.g. acceptance testing) Ensure a reliable way of flagging and correcting flight criticalpart quality problems Absent at GD Resulted in this completely preventable loss Conduct full scale tests to understand dynamic aspects (e.g.loads) of separation systems 34 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 35. The Engineering Challenge of Electrical Disconnects - Video35 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 36. Titan Centaur TC-6 Voyager 1 Category: Near Miss Problem: TC-6 Stage II oxidizer tankdiffuser lodged in a position torestrict flow to the engine (9/5/77) Why: Probably a breakdown infactory discipline (inspection paperprocessed in advance - subsequentfastener installation not completed) Impact: Stage II shut down 544ft/sec. slower, and 3,000 ft lower,than predicted, placing a hugeburden on the Centaur upper stageto compensate 36 2006 All Rights Reserved. Aerospace Engineering Associates LLC 37. Titan IIIE Stage II Autogeneous Pressurization System 37 2006 All Rights Reserved. Aerospace Engineering Associates LLC 38. Titan Centaur TC-6, Voyager 1 (contd) Fell to near bottom of tank Restricted N2O4 flow38 2006 All Rights Reserved. Aerospace Engineering Associates LLC 39. Titan Centaur TC-6 Voyager 1 (contd) Impact (contd): Favorable Earth-Jupiter geometry on 9/5/77 permittedCentaur to have enough propellant tomake up the shortfall Irony: Had the same failure occurredon TC-7 (launched 8/20/1977) insteadof TC-6, Centaur would very likelyhave run out of propellants beforeachieving the target Consequence: Voyager 1 would havefailed, and Voyager 2 could not havecompleted the Grand Tour ofJupiter, Saturn, Uranus, and NeptuneSources: NASA Lewis TC-6 Voyager Flight Report (undated) (NASA GRC Archives); Titan III E-6Stage II Investigation (undated) (NASA GRC Archives); Subject Matter Experts: (RichardGreenspun, Martin Marietta (ret.);Tom Chandik, General Dynamics);39 2006 All Rights Reserved. Aerospace Engineering Associates LLC 40. Titan Centaur TC-6 Voyager 1 (concluded)Lesson: If the hypothesis is correct, factory floor discipline broke down Human systems fail Theres a limit on what one can do to prevent it Doesnt mean one shouldnt try! Protect against these failings by: Training Audits Increasing awareness of consequences of carelessness Impact to company Impact to Nation Impact to individual Avoid blind installations as a matter of good engineeringpractice40 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 41. The Largest Disaster in the History Of Rocketry Baikonur Cosmodrome Russia,10/24/1960 Preps for first test flight of R-16 ICBM Program rushed to launch onanniversary of Bolshevik revolution(as a present for Premier Kruschev) Mitrofan NedelinR-16 ICBM Lead by head of the Soviet BallisticMissile Forces Marshal MitrofanNedelin 250 people on and around pad Viewing stand for visiting dignitariesDestroyed Pad and Memorial at Baikonur (Tyuratam) Unsafe design and undisciplinedprocedures caused 2nd stage ignition More than 120 people were killed Videoincluding Nedelin41 2006 All Rights Reserved. Aerospace Engineering Associates LLC 42. Observations Only one of the 35 mishaps analyzed (Atlas Centaur 24) had failure of a proper part as the cause! Programs doing good job of acceptance testing Therefore, conventional risk assessment based on piece part failure rates is, at best, incomplete The other 34 were caused by human error, management weaknesses, systems engineering shortcomings, etc., which are not easily modeled 42 2006 All Rights Reserved. Aerospace Engineering Associates LLC 43. Another Observation:Testing Deficiencies Had a Pivotal Role in 20 of 35 Cases Analysis/Modeling/System Testing to: Case Acquire Data to Qual/Accept Verify Operational Enable DesignHardware or S/W Functionality Atlas Centaur F-1? Apollo 13 POGO ? ? Apollo 13 ExplosionX Atlas Centaur 24 O O N-1O X Titan Centaur 1O Seasat O Atlas Centaur 62 X STS-51/TOS X Ariane 501 O LewisX SOHO X WIRE X Titan IVB-32 O Mars Climate Orbiter O Mars Polar LanderX X-43AX Helios O GenesisO DART X O-Omitted; X-Mis-performed; ?-Category Unknown 43 2006 All Rights Reserved. Aerospace Engineering Associates LLC 44. Observations (concluded) Programs that adopt a minimalist approach totesting are betting on the ability of the engineeringcommunity to foresee all aspects of systemperformance under all conditions This is a very risky bet!History demonstrates that tests frequently, if notusually, produce unexpected (and unwanted) results 44 2006 All Rights Reserved. Aerospace Engineering Associates LLC 45. What Can Be Done to Avoid Mistakes of the Past? Every project must implement a very specific set of principles which must be followed and rigorously enforced For example: Everything which can be tested on the ground will be All analyses must be independently verified and based on testdata where possible Heritage Systems: Any system flown successfully on an earliermission will be deemed unsuitable for flight until retested Enforced principles such as those above would have prevented many mission failures ! 45 2006 All Rights Reserved. Aerospace Engineering Associates LLC 46. Conclusions 46 2006 All Rights Reserved. Aerospace Engineering Associates LLC 47. Conclusions Most mishaps can be broadly attributed to human error, notrocket science Lack of complete understanding of how complex systems interact with each other Inadequate attention to every detail Flawed analyses or tests Improper use of heritage systems Flawed processes Flawed understanding of how software fails Reaction to budget or schedule pressure Imperfect management Often, a complex, subtle, sequence of events is needed If just one event in the chain were prevented, the failure would not have happened Must ensure quality in all the above areas Essential for mission success Over decades, the same root causes of failures appear repeatedly There are few new ones!47 2006 All Rights Reserved. Aerospace Engineering Associates LLC 48. Conclusions (contd) Why Dont We Learn From Past Mistakes? The lessons are learned mostly by the people who were therewhen the failure occurred With time, the keepers of the knowledge disappear Whats left is a diminishing, second-hand memory that also fades quickly Paper, or even electronic, lessons learned data bases (as good as theyre getting) are still insufficient by themselves to keep the memory alive Lack the live element to Reveal the nuances Fill in the details the official record omits Convey the passion Nothing beats talking to those who were there Basically, there is no universally successful approach to learningthe lessons from the past The business of transferring lessons learned is best done as a contact sport 48 2006 All Rights Reserved. Aerospace Engineering Associates LLC 49. P. O. Box 40448 Bay Village OH 44140 www.aea-llc.comJoe Nieberding, President Larry Ross, CEOEmail: joenieber@sbcglobal.netEmail: ljross1@att.netCell: 440-503-4758Cell: 440-227-7240 MISSION AEAs mission is to leverage the vital lessonslearned by NASAs spacefaring pioneers to strengthen the skills of todays aerospace explorers. 2006 All Rights Reserved. Aerospace Engineering Associates LLC 50. Appendix A: Glossary of TermsTermDefinitionTermDefinition Term Definition ACS Attitude Control SystemFODForeign Object Debris JPLJet Propulsion LaboratoryAdvanced Communications ACTS GAOGovernment Accountability OfficeJSCJohnson Space CenterTechnology Satellite AEAAerospace Engineering AssociatesGD General DynamicsKSCKennedy Space CenterAutomatic Determination and ADDJUSTDissemination of Just Updated GPSGlobal Positioning System LCCE Life Cycle Cost EstimateSteering Terms APLApplied Physics LaboratoryGN&C Guidance Navigation and Control LM Lockheed Martin APU Auxiliary Power Unit I&TIntegration and TestLOXLiquid Oxygen ATKAlliant Techsystems IC Integrated CircuitLMALockheed Martin Astronautics AV AeroVironment, Inc. IIPInstantaneous Impact PointLSPLaunch Service Provider Independent Program Assessment Microgravity Droplet Combustion BFCBetter Faster Cheaper IPAO MDCA Office Apparatus CAIB Columbia Accident Investigation Board IRUInertial Reference Unit MESMain Engine Start (Centaur) CONOPS Concept of Operations ISAInitial Sun Acquisition (SOHO)MMHMonomethylhydrazineDefense Meteorological Satellite DMSP ISSP International Space Station Program MO Mars ObserverProgram Independent Verification and ESREmergency Sun Reacquisition (SOHO)IV&V MOUMemorandum of Understanding Validation50 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC 51. Appendix A: Glossary of Terms (concluded)TermDefinitionTermDefinitionTerm Definition MS Meteoroid Shield (Skylab)S&MASafety and Mission Assurance SRRSystem Requirements Review MSFC Marshall Space Flight Center S/C Spacecraft SSME Space Shuttle Main Engine Science Applications International NACNASA Advisory CouncilSAIC SSTO Single Stage to Orbit CompanyNational Aeronautics and Space NASASDR System Design Review STSSpace Transportation SystemAdministrationNational Oceanic & Atmospheric NOAASESystems EngineeringTOSTransfer Orbit StageAdministration NRANASA Research Announcement SEB Source Evaluation BoardUAVUncrewed Aerial Vehicle Systems Engineering Management NTONitrogen Tetroxide (N2H4)SEMP USAF United States Air Force Plan OSPOrbital Space PlaneSLI Space Launch InitiativeVSEVision for Space Exploration P&WPratt and WhitneySOA State of the Art PDTProduct Development Team SOX Solid OxygenLongitudinal oscillation (as in POGO Space Plasma High Voltage POGOSPHINXstick not an acronym)Interaction Experiment RLVReusable Launch VehicleSRB Solid Rocket Booster RSRM Redesigned Solid Rocket MotorSRM Solid Rocket Motor51 2006 All Rights Reserved. Aerospace EngineeringAssociates LLC