Framework for Compliance, Verification, and Non-Conformance

George Angeli

LSST All-Hands MeetingBremerton

August 17, 2015

LSST Systems Engineering 2

Outline

8/17/2015

• Non-conformance framework (George Angeli)–Quality assurance at project and subsystem level–Change Control Board–Acceptance Data Package–Case studies

• Verification and Validation on LSST (Brian Selvy)

• Subsystem Acceptance and Verification in Commissioning (Chuck Claver)

LSST Systems Engineering 3

Overview

8/17/2015

• Basic procedures of Quality Assurance implemented by subsystems–Further refinement in progress (eTraveler, detailed NCR process)–Project QA should match in maturity the subsystems–Bottom-up and top-down procedures should meet in the middle

• Objectives–Establish a process for handling non-conformance at project level

•Well documented and uniform•Transparent and predictable for the project community•Do not interfere with current subsystem procedures

–Establish a process for documenting deliverables to and by the project•Uniform across all subsystems•Provide a well organized archive for the lifetime of the Observatory•Accessible for the project community and the operations team

LSST Systems Engineering 4

Quality Assurance

8/17/2015

• Well established regime of inspections and tests• Well defined responsibilities• Traceable and detailed documentation of deliverables and their

verification/certification• Transparent and consistent handling of non-conformance • Every subsystem has the responsibility to establish its own QA• Lower level plans (safety, QA, verification, review, etc.) should

conform with higher level ones • Project level coordination of deliverables

–To the Project by subsystems, and–By the Project to stakeholders/operators

• KEEP IT SIMPLE! (as much as possible …)

LSST Systems Engineering 5

The “Linear” Way of Acceptance

8/17/2015

Design Requirements

Documents

Interface Control

Documents

Verification Plan

...

Inspection

Calibration

VerificationActivities

Acceptance Data Package

Acceptance

Standards &Guidelines

Review Reports(when

applicable)

All requirements are met

Full compliance

LSST Systems Engineering 6

The “Linear” Way of Acceptance

8/17/2015

Design Requirements

Documents

Interface Control

Documents

Verification Plan

...

Inspection

Calibration

VerificationActivities

Acceptance Data Package

Acceptance

Standards &Guidelines

Review Reports(when

applicable)

All requirements are met

Full compliance

PSE witnessing

PSE collecting

• Project Systems Engineering understanding of verification and as-built component behavior facilitates system verification and commissioning

LSST Systems Engineering 7

The “Not-So-Linear” Way

8/17/2015

Design Requirements

Documents

Interface Control

Documents

Verification Plan

...

Inspection

Calibration

VerificationActivities

Acceptance Data Package

Acceptance

Standards &Guidelines

Review Reports(when

applicable)

Non-Conformance

?

Full compliance

LSST Systems Engineering 8

The “Not-So-Linear” Way

8/17/2015

Design Requirements

Documents

Interface Control

Documents

Verification Plan

...

Inspection

Calibration

VerificationActivities

Acceptance Data Package

Acceptance

Standards &Guidelines

Review Reports(when

applicable)

Non-Conformance

?

Full complianceChange

Control Process

Deviation note

LSST Systems Engineering 9

Non-Conformance Resolution

8/17/2015

• Using the Change Control Process is convenient– Proper people are involved – bring understanding of all aspects of the

project– Well established, exercised process– Well documented – web site captures process steps– Transparent and archived process

• Deviation approval process1. Non-Conformance Report – as soon as it is recognized2. Change Request

• Filed against requirements, as usual, but• Can be filed for a single component only• All relevant documentation posted in the DocuShare collection

3. Impact study, as usual4. Decision, as usual

• CCB is an advisory body• Project Manager disposes it, per the Project Execution Plan

Turnaround time limited by the impact study, not by the process

LSST Systems Engineering 10

Acceptance Data Package

8/17/2015

• Relevant requirements– Specifications and interface requirements– Non-conformance reports, change requests, deviation approvals

• Compliance certification– Verification Matrix– Verification and acceptance test plans and procedures– Qualification and/or acceptance test results– Calibration and certification requirements records– Material and/or process certifications

• Safety documentation and constraints• As-built technical documentation• Shipping, handling and storage plans• Operations, maintenance, and repair plans and manuals

LSST Systems Engineering 11

Case Study 1: M1M3 Acceptance

8/17/2015

• Managed by the Telescope & Site (T&S) team– Jacques Sebag, in particular

• All tests were done by Steward Observatory Mirror Lab (SOML) personnel–Witnessed by representatives of the T&S and Systems Engineering

teams• PSE was invited to all meetings with the vendor

– PSE supported the T&S team with analyses and trade studies• Documentation

– All data received from vendor posted on an open confluence page– All documentation collected on project DocuShare

• M1/M3 Cast Mirror Specification  • M1M3 Mirror Verification Matrix• M1M3 Mirror Acceptance Test Plan  • M1M3 Mirror final acceptance report  • LSST M1M3 Measurements of Prescription, Dimensions and Alignment  • LSST Final Acceptance Review 150410 compressed.pptx

LSST Systems Engineering 12

Case Study 1: M1M3 Analyses

8/17/2015

• Rebuilding the “M1M3 Null Test Data Reduction Pipeline” (Document-17912)– Laying the foundation of fully understanding the most critical vendor

acceptance test• Potential non-conformance: crow’s feet on M3

– In support of the T&S team, PSE evaluated the impact on ultimate metrics (Document-17171)

• Image depth• Ellipticity (single visit)

– Positive outcome cleared the way to acceptance

LSST Systems Engineering 13

Case Study 2: L1L2 Assembly

8/17/2015

• Early involvement of PSE – Invitation to all reviews (PDR, 75%, 90%, FDR)

• Verification/test plans• Compliance status

– Sharing data and analyses (FEA results, scattered light, interface volume, etc.)

• Potential non-conformance issues–Glass refraction index uniformity

• Joint optical analysis with LLNL cleared it –Surface treatment of carbon fiber

• Deviation request against an OSS requirement• Based on PSE scattered light analysis, the request is about to be approved

LSST Systems Engineering 14

Summary

8/17/2015

• It’s essential to involve PSE early in

– Verification planning

– Compliance assessments

– Verification activities

• System level non-conformance is handled through the CCB

process

• Acceptance Data Packages are expected to accompany every

deliverable

– System Data Package are built on subsystem Data Packages

– Subsystem Data Packages are built on major component Data

Packages; and so on …