NASA Assessment Card “One Liners”

• A Noncompliance or Observation is written during an assessment.

• Then we summarize it in a “one-liner” description on the card.

Description

NOTE: If writing an observation or non-conformance to anything other than AS9100C, please note the clause and the revision of the Standard, e.g., AS9100B, section 4.3 vs. 7.1.3.Requirement: standard or internal document, paragraph, descriptionDescription: describe in sufficient detail the non-compliance or observation, include description of objective evidence

One Sentence DescriptionSummary of

the card written

Details of the Finding

NASA Assessment Card “One Liners”

• The next charts show a sample of “modified” one-liners from the last two years on the topic of Document Control (for about 80 total assessments).

• One-liner summaries for “Contract Review” and “Procurement” are also included.

GSFC Supplier Assessments, “Analysis of Findings and Achieving Compliance”, C. Robinson & B. Brunello, October 14, 2009

Safety and Mission Assurance Directorate, Goddard Space Flight Center

Supply Chain Management Program

3

• Document Control – Findings, pages 1 through 14

• Contracts Review – Findings, pages 1 through 9

• Procurement – Findings, pages 1 through 8

Document Control – Findings 1

4

Procedure released without correct reference to associated form and procedure titles are inconsistent.

A contracted employee created his own document control system for his use of TDRS documents on the TDRS server.

Supplier does not have any of the industry specifications listed in the LDCM OLI process specification for dip soldering.

The Software Quality Reporting Work Instruction and Procedure does not clearly identify who is responsible for the control and reporting of the software assurance.Obsolete specifications are listed in both Supplier’s MAP and Supplier’s Program Implementation and Product Assurance Plan.

Non-existent procedures listed in the JWST Quality Program Plan

There is no linkage from the Quality Manual or document control procedures to the Configuration Management Procedure.

Supplier is using a new system to manage the training needs of their personnel (MyLearning); however, it conflicts with released procedure that requires the use of “Training Tracker”.The templates being used to aid the GIDEP documentation are not under document control nor are the use of these templates documented to assure a consistent methodology.Supplier does not have a released Corrective Action and Preventive Action procedure and two (2) others required by ISO 9001. [repeat from FEB 2004]Supplier stated they updated a procedure, when if fact it was not updated.

Supplier committed to have 6 common procedures for the 3 sites, but have not defined which procedures nor is ther a plan for completion.Uncontrolled version of Command Media on the shop floor.

Document Control – Findings 2

5

Two customer documents were found in hardcopy in an office with no notations that they were uncontrolled and should be used for reference only.Some obsolete forms are provided to personnel and others are not available.

There was evidence of gaps in the form control process, such as finding several forms that had no control information and a form in use that was different from the form controlled via a procedure.The Unit Development Folder’s “Conclusions” field needs to be described as mandatory in the template.

Supplier does not have a documented process to assure the procedures referenced in another procedure are current.

The JWST & ISIM QA Plans (controlled documents) have not been updated or revised to include current reference procedures.

The required procedure to control documentation is not consistently implemented, e.g., divergence from current practices, multiple document errors.Procedures are not maintained current and management has not provided an alternate interim process.

The Software Requirements Specification references out of date customer documents.

Engineering drawings are red lined without explanation or benefit of initials and date.

Printed management system documents not handled according to internal document control procedure.  

Forms called out in the Supplier Selection & Evaluation procedure have been replaced without updating the procedure.

Employee Training Planner does not reflect current practices.

Document Control – Findings 3

6

Supplier has been working to a CM Plan since NOV 2007 that has not yet been approved by GSFC (CDRL #003). (May 2007 Card 10)The latest revision of calibration standards are not maintained as required by Supplier procedure.

Document Control does not maintain a copy of NASA-STD-8739.2, .3, .5, and .8 which are required on the LDCM contract.

Three (3) of the five (5) System Operating Procedures have not been reviewed within the annual time frame as required by Supplier internal procedure.Unpublished documents are printed and used.

The corrective action response submitted to NASA was not fully implemented in the work instruction and in the Redline Log. The JWST work instruction was not fully updated to reflect the QA category and the form did not accommodate such. (NOV 2007 #’S 17 & 34)Documents continue to have obsolete references. (JAN 2008 Card 21)

The procedures to be used during the on-going transition of the software disciplines into a documented CMMI structure is not clearly understood.Approval of several procedures lags the release date by 2 to 5 years.

There is not a documented process to update the appendix to a procedure without revising the procedure.

A procedure was updated and released in July 2008 that referenced another document that has yet to be released.

Supplier Management Review, contains references to obsolete Divisions.

Documents of external origin are not fully identified and their distribution controlled as required by ISO 9001. (FEB 09 Card #’s 12 & 14)

Document Control – Findings 4

7

There is not a procedure for obsolescing documents within the system.

Project Working documents, under configuration control were identified with a revision status of “Draft”.

The list of documents referenced is incomplete and interrelationships to ISO 9001 / AS9100 are in error.

The document control procedures do not reflect current practices.

Supplier’s Quality Manual Appendix P does not identify all internal controlling documents that relate to AS9100 clauses.

Forms utilized by the Customer LDCM program are not under version control. (Customer Action)

FaCETS Procedures do not reference related Statutory and Regulatory procedures or related Supplier Policies and interfacing Procedures as required.The three year document review cycle required in Supplier Control of Operating Documents Procedure, was not practiced for "level 3" documents such as work instructions.Supplier does not have a documented QMS Waiver process and not all QMS deviations are receiving final approval

There are inconsistencies between the forms/form numbers and the Supplier database.

The Subcontract Administrator procedure for flow-down of requirements was cancelled and the current procedure could identified.

The risk reporting form is an uncontrolled form.

FMI packing instructions are not on controlled forms nor is their use directed by procedure.

Document Control – Findings 5

8

Quality Procedure issued containing obsolete and superseded requirements.

An outdated version of a Report of Safety or Health Hazard form was available for use in the work area not in accordance with P0010, Rev E (prevent the unintended use of obsolete documents). Engineering has not signed the log approval column per Supplier internal requirement for over 750 procedures.

Numerous obliterated records were found in the JWST Work Instruction processing log.

Revisions of reviewed Customer project procedures found in Supplier Policy and Procedures Manual do not match electronic versions found on Customer PAR Share Site.Forms in use did not always have an issue date or revision, as required by internal procedure. 

Insufficient review and maintenance of key documented procedures allows for ongoing divergence between documented procedures and actual practices. Supplier personnel, are working to an unreleased draft document, with a Process Tailoring Request which had incomplete history log.Obsolete documents were found in a Controlled Book, posted in a Gowning room, and in the 4050 Metrology and a move log was not being used.Procedures are not maintained current or audited for accuracy.

A relevant version of P007 is not available to employees as required in P0010, Rev E (ensure that relevant versions of applicable documents are available at points of use). Supplier procedure 200-ENGR-100, Para. 6.4.3, requires documents be stored in “Supplier Vault”. No requirements for document identification or formal change control are documented.FaCETS Construction Quality Control Plan has no objective evidence of periodic review by the Program Manager as required by Supplier procedures.

Document Control – Findings 6

9

Hyper-links within several procedures did not work, pointed to a non-existent procedure, or linked to a different procedure than listed.CCB Record 104E did not identify the original or updated version of the software being changed.

The Solder Instructions states referenced documents are only applicable if they are identified within the body of the procedure, which they are not.Supplier does not have a process to control the distribution and archiving of customer or external standards as required by ISO, AS, and their own procedure.Some reference documents listed in the CM Plan for GOES-R are obsolete or not applicable and an applicable document was not referenced.Various Supplier procedures still contain obsolete references.

Document referenced in JWST 1104 does not exist.

Requirements Traceability Matrix found in Software Requirements Specification missing a requirement and out of sync with DOORSSupplier has two VIIRS Interface Control Documents each with the same number and revision, but different dates and content

The Engineering Design Review Process references a number of internal procedures which have been superseded by new documents.Civil Servants use revision C of the GOLD Rules Standard while contractors only have revision A, and the current D revision is not on GDMS.No objective evidence that documentation is reviewed every 2 years in accordance with Supplier’s Configuration Management and Documentation Control Procedure.The Peer Review process form is not complete and referred to by different names in different documents.

Document Control – Findings 7

10

Hardcopy documents do not have suitable identification to indicate that they are uncontrolled.

The online command media does not show the most current procedure due to faulty hyperlinks.

Supplier does not have a process to assure documents are maintained current with higher-level directives.

Supplier does not maintain a master list of the documents of external origin, nor do they have a method for ensuring that they are using the applicable standard.Nineteen (19) procedures were reviewed which engineers were using, three (3) of the nineteen (19) had the incorrect revision.

Obsolete documentation is available for use on the Supplier engineering drive, e.g., the Design Engineer responsible for JWST program maintains and uses a file that contains the JWST SOW that was superseded in July 2008.Manual changes made to an Engineering Change Request were not done per Supplier procedure.

Supplier is using a Reliability Program Plan that was created for another operation and does not reflect current practices.

A procedure was released 18 months ago and Work Instruction referenced in the procedure has not yet been released.

A procedure was released with a non-existent and incorrect reference document.

Supplier on line system did not contain the latest SOW.

Obsolete (EHS) Command Media was found in the High Bay and the High Bay Gown Area

Individuals are assigned responsibility for given functions, however, approval of the documents in the electronic Command Media system is done by a different individual.

Document Control – Findings 9

11

There was no evidence of a current Program specific approved materials list.

An obsolete procedure was found on the floor in the GLAST Gowning Room.

Mulltiple procedures/processes do not reflect actual practices with respect to the control of non-conforming product and corrective/preventive actions.Software Quality Assurance (SQA) metrics are not being collected according with the multiple documented procedures.

Document Control procedure does not address the review of procedures.

Plans in place to manage the impact of revisions to industry standards, but not realized at this time.

Accuracy and Maintenance of Organizational Chart documents are a concern.

There is not a process in place to proactively track checked out documents.

The procedure for Contract Administration did not reflect current practices for Contract Requirement Flow-down.

Supplier does not have a process that identifies revisions to industry standards used by a department. (SEP 07, Card #05)

The SEMP has conflicting and missing revision dates.

Eight AT’s containing in-house approved red-lines were reviewed and found to have legibility issues due to line-outs, overwriting, and 2 entries made in pencil. Several internal procedures released by Supplier had errors.

Document Control – Findings 10

12

Supplier has developed both a Hazardous Material Control Plan and an Emergency Action Plan, both identified as “Rev 0”, but neither have a document number nor are they under document control. (FEB 09 Card # 23)The approval date on several Supplier procedures is after the effectivity date.

System Engineering Management Plan (SEMP) for Glory APS does not have a release date in the official on-line version. (May 2007, Card 03)Supplier Project Policy and Procedures Manual contain two (2) Table of Contents - one dated 2/25/08 and a second dated 4/15/05.The cover sheet for the SUVI Software Management Plan records a date in the upper right hand corner, below the document number, which the software engineer thought was the release date, but is the date the cover sheet was routed for signatures. No formal documented requirement exists to review Supplier Practices

The document providing the status of calibrated items requires a new revision for each update. Supplier does not follow this self-imposed requirement. Procedures used for Interdivisional Work Authorization (IWA) are not cross referenced and the corporate IWA procedure is not under configuration control.The procedure supporting the 5S process is not reflected in the released procedure. (JAN 2008 Card 49)

Signature Examples Sheet, QF-824-1 has not been maintained. Only one of four employees required for recently approved documents are on this list.The NuSTAR Spacecraft Bus System Requirements was released by Customer without an approval signature on the document. (Customer Action)The System Safety Engineering procedure references a non-existent MIL-STD.

Supplier procedure does not clearly specify who approves initial release of documents.

Document Control – Findings 11

13

Quality Assurance (QA) is required to approve Major and Minor Changes to manufacturing work orders, but the form does not have a place for a QA signature.A procurement procedure used by Supplier, but controlled by another Supplier operation was not reviewed every two years as required.The NuSTAR Contract Document Subcontract document contains fields that were not completed and missing dates. (Customer Action)Control of Supplier Processes and Procedures is not on-line making distribution and maintenance of this media difficult.

The Government-Industry Data Exchange Program (GIDEP) Notification document did not reflect current practices.  

The Software Management Plan/Software Development Plan document is not being properly edited.

The Enterprise Quality Management System Manual does not follow the document structure outlined in EI-05-06.

Employee names are given in an organizational chart rather than positions which makes them hard to be maintained current.

The Supplier’s Project Management Plan’s associated document links are not functional.

Hard copies of Desk Instructions do not indicate that the appropriate document control database is to be checked for the correct version of the Instruction prior to use.Some procedures do not provide sufficient direction or are too new to validate compliance.

QMS Implementation MAP and other documents contain incorrect references.

On the JWST project the risk management system does not have the proper amount of access control.

Document Control – Findings 12

14

The personnel signing off documents are not consistent with the procedure.

Engineering Documents do not have a template for consistency between programs.

Supplier does not have a process to assure that the Program Directives and Procedures & Processes are updated as required.

The process for how customer satisfaction data is gathered and utilized is not documented.

Document reviews that impact the entire organization are on occasion being reviewed by a single organization.

Responsibility for generation, approval, change control, and storage and retention of Work Instructions is not explicitly defined.

Supplier personnel have been verbally instructed to follow a draft procedure for the movement of government property.

Supplier does not require suppliers of standard parts to be on their Approved Supplier List nor do they have a process of periodically reviewing the suppliers performance for establishing the level of necessary controls.Seven examples of current Supplier procedures and forms that do not reflect current practices were identified.

Supplier has released procedures in the last few months impacting infrastructure. Supplier should consider adding an implementation date for new releases.A master copy of a Business Instruction was not contained in the required file, but was available on-line.

The Work Instruction Configuration Control Approval List is not reviewed by functional organizations for accuracy and adequacy.

Laboratory level documents are not reviewed at planned intervals to ensure continued suitability. Space level documents, however, are reviewed on a three year cycle.

Document Control – Findings 13

15

The Quality Manual was not updated with the most current AS9100 certification dates.

The on-going process of creating business instructions for software process from legacy documents leaves two versions of some processes in use which may cause confusion for software developers.The text in a Mission Operation procedure is not in agreement with the exhibits contained within the procedure.

STEREO SQA Plan needs to be updated with current process.

Forms referenced by procedures are not identified with a specific number or revision.

Supplier has an Operating Procedure identified as a Work Instruction.

The Procedure/Document Control system is cumbersome and difficult to locate applicable procedures.

Uncontrolled documents were in use on the manufacturing floor.

Supplier personnel were not able to readily access the pertinent procedures in several instances.  

Numerous typographical errors found on various Stoplight Chart Data Entry and History Sheets.

Obsolete forms are being used and in one case the current form was not available on the Supplier Oracle database.

QMS documents not periodically reviewed for currency and accuracy.

Document control revision access database should be limited to current revision.

Document Control – Findings 14

16

Flow down of customer requirements to Supplier is not timely.

Two obsolete versions of Supplier, Assembly Weld Schedule were found at different work stations in the work area. 

Stamp Control Procedures are inadequate.

The Supplier on-line document file contained a ‘preliminary’ WISE Safety and Mission Assurance Plan; not the current one dated 7/18/2005.Evidence of submittal and of customer approval could not be located for JWST SDRL items.

Obsolete Specifications used to create Supplier procedures.

Lack of definition of Quality requirements in subcontracts records.

Supplier Division Business System Manual’s scope is not appropriate

Supplier Division Business System Manual does not apply to all sites.

The Glory Configuration Plan is outdated, cage code, etc.

Software Management Plan has conflicting references.

Contracts – Findings 1

17

The IBEX MPAIP, PAIP, or SSPP do not have evidence of GSFC SAM or

Quality objectives are not established as part of the quality management system in accordance with Supplier Quality Assurance Manual and the ISO 9001:2000 standard.Measurable objectives were not clearly established nor communicated within the Supplier organization as part of the Customer Business Management System. Quality objectives are supported by measures (metrics) that are routinely collected, reported, and utilized except for the "Empowerment" objective for which performance or achievement is not measured. Key management responsibilities and authorities were not fully established and effectively communicated as part of the management system. Reporting relationships of the Quality Assurance function are sound in practice, but are not clearly and consistently defined in the Quality Assurance Manual.Insufficient frequency of QMS reviews and number of management participants to ensure continued adequacy and effectiveness of QMS.The DPAS Rating is missing on the Materiel Contract Brief for the Glory APS contract.

The Supplier Statement of Work does not list any applicable documents to be used to meet the contract requirements.

DPAS rating not included in purchasing or major subcontracts.

Customer should respond to a delinquent Class 1 change on the GLORY program. [Customer Action]

Quality Management System not implemented as contractually required for NASA projects (e.g.,MMS, Juno )

Customer is responsible for ensuring performance of all necessary actions for effective contracting, ensuring compliance with the terms of the contract. The engineer responsibility for approval of CDRLs should be discontinued. (Customer Action)

Contracts – Findings 2

18

The Supplier did not flow down the DPAS Rating of DO-C9 on the JWST contract. (Customer Action)

Supplier is contractually required to have a QMS compliant with ISO 9001 on the IBEX and GOES-R projects. A waiver for relief has been submitted for GOES-R but not for IBEX.Contractual requirements were not fully imposed on a major subcontractor.

Acknowledgement and acceptance of Change Orders by a sub-contractor is questionable.

There continues to be aged SDRLs on VIIRS, CrIS, and other NPOESS payloads contracts that require approval by Supplier.

NASA-STD-8739 series had not been fully encompassed in the workmanship manual before completion of the SDO program and a contract waiver to this requirement has not been submitted.NPP - There are 11 aged SDRLs on VIIRS contract that require approval by Supplier. (repeat issues from the following assessments Supplier - JUN 07 #14 and Supplier - JUL 08 #8)A DPAS rating of DO-C9 has been imposed on a general contract for services applicable to numerous projects. (This is a Customer Action)The Order Entry & Acknowledgement block on the two (2) contracts reviewed was not completed as required by Supplier procedure.Many documented software processes required by the Subcontract Mission Assurance Requirements Document (SMARD) are missing from VIIRS Software Development Plan.Supplier contractually accepted specification revisions that do not exist.

Requirements in the Contract between Customer and Supplier need to be better understood by the organizations executing the work.Customer, Safety and Mission Assurance Plan requires minimum content for the Safety and Mission Assurance Plan The minimum content has not been addressed in Supplier.

Contracts – Findings 3

19

Contract review for GOES-R contract has yet to be performed as required by Supplier procedures.

Flight 1 software is currently in maintenance (already developed) and Supplier does not know the Software Criticality Rating requirement which is necessary prior to software development. [This is a Customer issue]The Safety Plan references a non-existent CDRL.

The LRO SOW provides an option to use an ESD standard they identified that was obsolete 5 years before the LRO SOW. (This is a Customer action)Insufficient Contract Requirements (including Mission Assurance requirements) have been imposed on Supplier.

Old and non-NASA specifications are contractually imposed on the Supplier contract. (Customer Action)

MAP documents include inconsistent and incorrect references to the applicable reference documents.

Customer has not responded to a Mission Assurance Requirements compliance matrix Supplier submitted over a month ago. [Customer Action]Supplier listed an obsolete document in a SOW Table, but never imposed the document. This is an Supplier action.

NPR Procedural Requirements (NPR) 7150.2, Software Engineering Requirements, was not flowed down to Supplier in their Statement Of Work (SOW) (Customer Action).The ABI Optical Bench contract is not defined.

Supplier submitted the Hydrazine Propulsion Module (HPM) Program Plan to their customer but it has not been approved within the 30 day SOW requirement. (This is a Customer action)Customer is not completing the Supplier Customer Satisfaction Card. – This is a Customer Action.

Contracts – Sample Findings 4

20

The Manufacturing Quality Plan for Orion TPS does not comply with FMI requirements.

Customer Purchase Orders impose a Blanket Purchase Agreement (BPA Terms and Conditions), however, Supplier is unable to objectively identify relevant requirements and which articles on the BPA are applicable.Customer Contractual requirements noted as “NA” on Contract Review Checklist when specific requirements were flowed.

Supplier is not obtaining Contracting Officer’s written approval prior to bringing contractor or leased property on site.

ESD Training is performed to Supplier criterion that has not been approved by NASA as an equivalent to ANSI/ESD S20.20-2007.

The review and implementation of contractual requirements is not sufficient to ensure that the calibration system is compliant with the GPM contractual requirement: Supplier uses a less stringent calibration requirement.The Supplier Quality Assurance Plan is not maintained current with contractual requirements.

Supplier continues to work to the initial proposal on the EDO contract and are not fully compliant with the subsequent revision to the SOW which they contractually accepted. Supplier review of customer requirements was not complete, and Supplier did not fulfill a customer’s GIDEP requirement for the GOES-R/ABI program.Customer relieved Supplier of property tracking activities approximately 4 years ago, however, the contract statement of work has not been amended to reflect the change. (Customer Concern) Supplier does not have a reliable process for identifying applicable contract requirements. Methodology needed to flow down contractual requirements to the different organizational disciplines is lacking.Personnel are not currently certified to all of the Supplier Workmanship and Soldering requirements.

Supplier committed to comply with the requirements listed in RFP but did not have all the applicable documents, e.g., a cited statement of work.

Contracts – Sample Findings 5

21

Customer’s released Interface Requirements and Control Document on JWST’s SIDECAR contains TBD’s and TBR’s. (Customer Action)A contract requirement on a shipped LDSAT product was not met and Supplier has not obtained a waiver. (Ref SEP 08 Card #10)

Contract Compliance Matrix was not filled out completely.

It is not evident that the review of the letter contract for the LDCM project included the referenced statement of work and specifications.Contract review process did not capture all changes in a revised LDCM Statement of Work (SOW).

Customer Imposed a “TBD” document on JWST Micro-Shutter Assembly Performance Specification in June 2008 and 7 months later the specification is still “TBD”. – This is a Customer ActionSupplier accepted and worked to undefined contractual requirements contrary to contract review requirements, e.g., JWST Actuator contract with “TBD” requirements.All appropriate project engineering stakeholder organizations did not review requirements before committing to supply product.

Technical agreement on changing a design requirement was not incorporated into approved contractual documentation, e.g., JWST actuator power consumption requirements changed the wattage requirement based on conversation/e-mail. DOD-STD-2167(A) is an obsolete standard and no contract amendment was found to identify the appropriate standard.

Supplier does not have copies of all the specifications called out in contracts or in their internal documents.

Contract review process is not fully effective (e.g., acceptance of quality management requirements without the ability to effectively implement)

Contracts – Sample Findings 6

22

It is not evident that Customer approval was issued and received for the SDO S-Band Transponder/Diplexer Performance and Design Verification Test Plan as required by the SOW. – This is a joint Supplier & Customer ActionCustomer has not issued Supplier a SOW that has been in process since 9 MAR 2006.

(JOINT ACTION) An instance of insufficient clarity, understanding, and resolution of customer requirements in contractual documentation.  Requirements that became obsolete in 1999 were included in the contract initially signed in JAN 2002.

The contract Compliance Matrix was not always reviewed prior to contractual acceptance to assure contractual reconciliation.

The revision of the Customer Statement of Work issued to Supplier is identified as both revision 3 and revision 4.

Subcontracts Management flowed down an obsolete and inadequate quality system to two major subcontractors.

Effective contract review practices were not reflected in the contract review procedure.

The improvement of the Contract Review process is under way, but not fully implemented.

No evidence of Product Assurance review of the RBSP Procurement Performance Assurance Requirements (ProcPAR).  

In practice actions arising from quote review process are documented and maintained, but are not required by internal procedure for engineering.  The Quality Policy Manual states that undefinitized contract actions are not to be open longer than 180 days; the LDCM contract did not meet this requirement as it is still undefinitized.

Contracts – Sample Findings 7

23

Material and Purchasing contractual requirements are not reliably included in purchase orders. (Supplier Purchasing Flow down)

Customer SOW to Supplier needs a through review in all areas, especially reliability and SDRL List. This is an Customer action.

Customer has not provided Supplier with the current Interface Control Document for the VIIRS instrument. [Customer Action]

VIIRS Operating Plan is mute on review/approval authority for major subcontractors SDRLs.

CDRL’s were 3 months delinquent in JAN 08 and today CDRL #74 on the LDCM contract is 4 months delinquent. (JAN 2008 Card 30)Monthly status reports required per NIRCam SOW/SDRL are not fulfilled.

Product returns perceived by the customer as not being adequately addressed.

There are 5 delinquent GLAST CDRL with the oldest one being due 13 JUL 2007.

Supplier informed Customer in FEB 2009 that Supplier is not compliant with the contractual calibration requirements. Customer has not responded. (Customer Action)Contract, Mod 16 signed by the Government 4/29/2009 has not been signed by Supplier.

Customer approval system is not timely and Supplier doesn’t have visibility into changes impacting CrIS.

Action not taken to amend/remove a Statement of Work (SOW) reporting requirement to reflect actual arrangements for communicating with the customer.  Customer is not responding to submitted CDRL’s in a timely manner. One has been waiting approval since August 2004. (Customer Action)

Contracts – Sample Findings 8

24

Supplier customer did not flow down NASA-Std-8739.3 which restricts the use of non-rosin core solder unless prior approval is received from NASA. (Customer Action)Customer instructed Supplier that the submittal of single and double sided coupons to GSFC for approval was not required on the IBEX mission.Customer is still not providing timely responses to Supplier contractual requests. (Customer Action)

The System Safety & Mission Assurance Plan was written to reflect a deviation that was not granted and should be updated.

Customer is not providing a timely response to their supplier on various waivers and government property requests.

Management direction to establish Internal Audit procedure not evident, and the performance of planned internal auditing not sufficiently resourced at the Supplier. Supplier does not obtain contractual concurrence if they implement a requirement differing from the one listed on the contract.

The Product Quality Assurance Digest for both GSE and Flight list an obsolete specification not NASA-Std-8739.3 as required by the MAR. (Supplier Internal Flow down)Supplier uses the J-Std-001 specification for GSE, but this standard is not listed as an applicable document in the Customer workmanship control plan. (Customer Internal Flow down)Customer non-responsive to SDRL submittals.

Incomplete review / clarification of SDRL requirements revised by customer communication.

Rotation fixture, serial #1 for primary segments has an expired calibration sticker.

Product integrity is not assessed when equipment is found to be out of calibration. Calibration Control not in compliance with ISO 7.6 requirements.

Contracts – Sample Findings 9

25

Customers allowed Supplier to use a calibration standard that became obsolete 4 years prior to their respective approvals.

Customer listed an obsolete document as a viable specification for calibration. – This is a Customer action.

Purchasing and Supplier Control – Sample Findings 1

26

Supplier accepted an inspection document from a supplier that had an entry that was overwritten.

LDCM has a “DO” Defense Property Accountability System (DPAS) Rating that is flowed to Supplier but is not flowed by Supplier to their suppliers. Although the auditor was informed that EPLS was verified, there is no evidence to show that it was done or required. The EPLS was not used for supplier approval.Specific criteria for selection, evaluation and reevaluation of Suppliers has not been shall be established

The Approved Supplier List (ASL) does not include the scope of each supplier's approval as required by AS9100B, section 7.4.1.a).Minimal Supplier on-site surveys are performed and Supplier survey self evaluation survey forms are not critically reviewed by Supplier.Supplier Performance is not in the Approved Supplier List (ASL), as required by Supplier Quality Management document.

Supplier does not have a process of evaluating and re-evaluating suppliers as required in both ISO 9001 and AS9100. (Ref SEP 08 Card #10)A SCAR issued against a supplier was closed prior to verification of corrective action.

Records of supplier evaluation are not documented as required in AS9100

No objective evidence of the review of supplier’s rating systems.

Complete Vendor Ratings are performed on their three major suppliers, but are not fully performed on the remaining 147 in accordance with internal procedure.The Supplier Quality Subcontractor Information Database (QSID) contains the master list of Approved Suppliers, but does not provide scope of approval or indicate where it is accessed.

Purchasing and Supplier Control – Sample Findings 2

27

Two suppliers of calibration services were not included in the Supplier Approved Supplier List.  

Supplier does not require suppliers of standard parts to be on their Approved Supplier List nor do they have a process of periodically reviewing the suppliers’ performance for establishing the level of necessary controls.Records of compliance to Supplier Vendor Evaluation procedure could not be provided.

Supplier does not require full compliance with the SOW issued to a subcontractor on the JWST program and the SOW is not fully compliant with their contract. (MAR 2009 Assessment of TMT Card #05)Vendor Control Procedure, is available as a draft, but is not released. [repeat from FEB 2004]

A major subcontractor supporting JWST has a supplier survey that has been delinquent for over seven (7) months.

In the Acquisition Material Management System, there was a supplier listed as approved for Flight in one screen and not approved for Flight in another screen.Subcontract Administrators are responsible to manage the delivery of SDRL’s. Two Administrators that were interviewed did not know, nor had access to, the documents that indicated their status.Supplier does not have a supplier evaluation/rating scheme for product suppliers.

Out of a sample of 14 suppliers, two had delinquent surveys and two had incomplete supplier checklists.

Vendor Performance metrics are not maintained and reviewed by Supplier.

Divergence in processes between two divisions of a supplier for the management of critical shared suppliers. 

Supplier purchasing does not have a supplier rating system.

Purchasing and Supplier Control – Sample Findings 3

28

33% of the Supplier vendor folders were reviewed and all were missing several required documents.

Purchasing agent was not aware of approved supplier list in the Program Management Plan.

Supplier rating is not in compliance with internal procedures. Suppliers with no activity automatically default to 100% supplier rating after six (6) months.Supplier purchased, received, and accepted flight hardware from a supplier that was not approved for flight hardware.

A supplier currently being used by Customer, is listed as unapproved on the Customer Approved Supplier List.

No category that describes the standards compliance expectations for calibration service providers, i.e. A2LA, ISO/IEC17025 or Z540.- requirements.Approved Supplier Master List incomplete per internal procedure.

A change in quality rating of a supplier was not incorporated in the electronic database used by the purchasing agent in a timely manner (delay of approximately one month). (CUSTOMER/NASA) NASA requirements for record retention and for Lifting Devices and Equipment not flowed down to Supplier for the NASA RBSP Mission.  Supplier does not have a continual process for addressing suppliers below the 80% rating threshold, only a semi-annual re-evaluation.P0029, Rev B Purchasing (Supply Chain Quality) is outdated and inconsistent with lower tier documentation (W0229).

Supplier uses ‘vendor’, supplier’, and subcontractor’, interchangeably in the RBSP matrix which may lead to a lapse of requirements flow down by instrument providers.A supplier for organizational environmental health and safety management disposal was not identified in the ASL.

Purchasing and Supplier Control – Sample Findings 4

29

Supplier Management does not have procedure for nor took action on a major supplier with an unsatisfactory rating.

Two CrIS sub-contractors have QMS’s that have been identified as needing improvement for 2 years.

The original engineering approval of a supplier for flight hardware, including restrictions, are archived and not included in the current evaluation information.One out of a sample of eight (8) approved vendor survey folders did not satisfy Supplier requirements, yet was approved.

Requirements that establish Quality and Delivery requirements are identified as “should” statements as opposed to “shall” statements.Flight material and services suppliers are not all properly identified on the Supplier Approved Supplier List.

Only a partial record of supplier evaluation results are maintained.  

Supplier does not include delivery performance in their supplier rating system. The delivery rating is maintained in a separate data base.Supplier considers suppliers with an 80% on-time delivery rating acceptable.

Purchase Requisitions were incomplete, and no Inspection requirements identified.

The ASL data supporting supplier approval needs to be readily available to SQA.

A procedure that supports the activity of sending Supplier Report Cards to suppliers could not be found.

There was no documented method to notify buyers that a product was critical and required selection from the more stringently approved suppliers.

Purchasing and Supplier Control – Sample Findings 5

30

DPAS Rating is not flowed down to supplier as required by the regulation.

Suppliers do not record the electronic measurement instrument used to accept product and a purchase order to a mechanical supplier listed the old quality clauses.No objective evidence was available to substantiate procurement metrics as required by internal procedure.

The seven Supplier Quality System Questionnaires reviewed had some approval errors and the acceptance criteria could not be defined.Purchase Orders from Supplier to their calibration supplier do not impose a calibration system requirement.

Purchasing procedure requires flow down of a specification that is in conflict with both JWST & LDCM OLI requirements. (FEB 09 Card # 19)Supplier records used for evaluation were not consistently properly controlled.

The supplier list was reviewed and several supplier assessments are delinquent in accordance with the supplier evaluation process.Purchasing procedure CSOP-07-P does not describe what actions are required by the purchasing organization in order to flow customer specific Government DPAS DX and DO rated requirements to the supplier.The internally required calibration specifications are not imposed on the outside calibration suppliers.

Supplier Interoffice Correspondence providing the re-delegation of signature authorization was dated 27 February 2008 and has not been issued to date in 2009. This does not comply with internal procedural requirements.A record used in the Mechanical Systems Division to evaluate suppliers lacked clarity and inconsistent with the Approved Supplier List.If the standard Supplier Quality Notes do not encompass specific contractual requirements, the requirement is not communicated to the supplier.

Purchasing and Supplier Control – Sample Findings 6

31

DPAS ratings are not on purchase orders and there is not a process to verify that suppliers have not been Debarred.

DPAS Ratings are not communicated to Purchasing or included on Purchase Orders.

Suppliers do not consistently record the measurement instrument used to accept product which is required by Supplier’s Quality Assurance Manual.Supplier does not require their calibration supplier to be compliant with ISO 17025 which is required on their JWST contract.

A purchasing agent uses Command Media stored on his computer, not the official Command Media that is maintained on-line.

The MMS MAR requires calibration to ISO 17025 yet Customer allows the Supplier to calibrate to ANSI/NCSL Z540-1.

Purchasing procedure requires flow down of a specification that is in conflict with both the JWST & OLI requirements.

Purchase Orders to calibration suppliers do not require the supplier to comply with ANSI/ NCSL Z540-1 which is a customer requirement.Quality Management System requirements are not contractually included in Supplier purchase orders.

Purchasing Requisitions are issued with missing authorized signatures and information.

One (1) out of three (3) Purchase Orders was missing the Quality Clauses that were imposed on the Purchase Request.

Supplier self evaluation surveys do not assure compliance with contractual specifications.

Supplier is non-compliant with ISO 9001 which is a contractual requirement from Customer.

Purchasing and Supplier Control – Sample Findings 7

32

Concern with flowdown of requirements to supplier/partner.

System Safety assessments of suppliers are not performed.

Purchasing procedures do not accommodate official direction from Quality.

Quality is consistently identifying errors in Purchase Order Notes that require correction.

Special paper required for issue of documents is not identified by part number in the procedure that governs its use.

Supplier Quality Codes imposed on suppliers do not accommodate the GLORY requirement of ANSI/ISO/IEC 17025.

Supplier, competitive, and potentially TDRS sensitive procurement data is stored on company laptop hard drive. This is in conflict with the TDRS contract.The Vendor Rating Procedure no longer reflects the current practices which are undergoing improvement.

The methodology for the review and acceptance by reliability and components engineering of EEE parts is outlined in a PowerPoint file; not documented in a controlled procedure.An inspector accepted the packaging of a component without being able to identify the specific packaging requirements.

JWST ASIC connectors were received, inspected, and accepted without fully complying with the purchase order requirements.

Supplier only has a test program for OMPS at receiving inspection for prohibited materials.

Supplier does not provide criteria to engineers who are responsible for determining if a purchased item should be inspected.

Purchasing and Supplier Control – Sample Findings 8

33

There is no objective evidence that a supplier complied with the special shipping instructions on two Diplexers.

Receiving Inspection records state that all dimensions are inspected, however, close tolerance dimensions are not verified.

Supplier QA accepted a material certification that was compliant with the alloy designation, but the drawing required bar stock and the vendor used plate stock.Statistical sampling at receiving inspection is written based on a specification that was superseded in 1964. Also, the procedure was not communicated to receiving inspection who uses a non-controlled reference table.Receiving Inspection of fasteners for the Juno Project did not adhere to the requirements of Supplier Work Instructions.

Source Inspection acceptance of electrical components is being performed by people not certified in the discipline.

Receiving Inspection does not appear to be sufficiently staffed as 50% of the 70 receipts have been waiting for over 3 months.

Tracking multiple processing of batches at a division of a Supplier is not as robust as in the Supplier’s main facility where it is accommodated by the SAP system.The Supplier Metrology lab procedure does not reflect the actual practice of screening and accepting defective lots.

Two Division is only notified of nonconformances for shared suppliers when the another division believes it affects the the other Division’s product lines.  Purchasing was not informed when Operations initiated corrective action with a problem supplier.

JWST Deviation/Waiver Requests final disposition information completed by Supplier does not provide adequate information to fully document the rationale for the disposition decision.The Laboratory level Vendor rating system has subjective components that could invalidate any potential benefits.