10.4 Industrial Safety and Quality Assurance

¶The industrial safety programs in place at NASA and its contractors are robust and in good health. However, the scope and depth of NASAʼs maintenance and quality assurance programs are troublesome. Though unrelated to the Columbia accident, the major deficiencies in these programs uncovered by the Board could potentially contribute to a future accident.

¶Industrial Safety

¶Industrial safety programs at NASA and its contractors— covering safety measures "on the shop floor" and in the workplace – were examined by interviews, observations, and reviews. Vibrant industrial safety programs were found in every area examined, reflecting a common interview comment: "If anything, we go overboard on safety." Industrial safety programs are highly visible: they are nearly always a topic of work center meetings and are represented by numerous safety campaigns and posters (see Figure 10.4-1).

¶Figure 10.4-1. Safety posters at NASA and contractor facilities.

218

¶Initiatives like Michoudʼs "This is Stupid" program and the United Space Allianceʼs "Time Out" cards empower employees to halt any operation under way if they believe industrial safety is being compromised (see Figure 10.4-2). For example, the Time Out program encourages and even rewards workers who report suspected safety problems to management.

ASSERTIVE STATEMENT

¶OPENING Get person's attention.

¶CONCERN State level of concern.

¶Uneasy? Very worried?

¶PROBLEM State the problem, real or perceived.

¶SOLUTION State your suggested solution, if you have one.

¶AGREEMENT Assertively, respectfully

ask for their response. For example: What do you

¶think? Don't you agree? EVERY EMPLOYEE

¶When all else fails, use "THIS IS STUPID!" to

HAS THE RIGHT

¶alert PIC and others to potential for incident, injury, or accident.

¶c 1999 Error Prevention Institute 644 W. Mendoza Ave., Mesa AZ 85210

¶Figure 10.4-2. The "This is Stupid" card from the Michoud Assembly Facility and the "Time Out" card from United Space Alliance.

¶NASA similarly maintains the Safety Reporting System, which creates lines of communication through which anonymous inputs are forwarded directly to headquarters (see Figure 10.4-3). The NASA Shuttle Logistics Depot focus on safety has been recognized as an Occupational Safety and Health Administration Star Site for its participation in the Voluntary Protection Program. After the Shuttle Logistics Depot was recertified in 2002, employees worked more than 750 days without a lost-time mishap.

¶Quality Assurance

¶Quality Assurance programs – encompassing steps to encourage error-free work, as well as inspections and assessments of that work – have evolved considerably in scope over the past five years, transitioning from intensive, comprehensive inspection regimens to much smaller programs based on past risk analysis.

¶As described in Part Two, after the Space Flight Operations Contract was established, NASAʼs quality assurance role at Kennedy Space Center was significantly reduced. In the course of this transition, Kennedy reduced its inspections – called Government Mandatory Inspection Points – by more than 80 percent. Marshall Space Flight Center cut its inspection workload from 49,000 government inspection points and 821,000 contractor inspections in 1990 to 13,700 and 461,000, respectively, in 2002. Similar cutbacks were made at most NASA centers.

¶Requirements Document (also called the Mandatory Inspections Document). United Space Alliance technicians must document an estimated 730,000 tasks to complete a single Shuttle maintenance flow at Kennedy Space Center. Nearly every task assessed as Criticality Code 1, 1R (redundant), or 2 is always inspected, as are any systems not verifiable by operational checks or tests prior to final preparations for flight.

¶Nearly everyone interviewed at Kennedy indicated that the current inspection process is both inadequate and difficult to expand, even incrementally. One example was a long-

¶TIME standing request to add a main engine final review before transporting the engine to the Orbiter Processing Facility for installation. This request was first voiced two years before the launch of STS-107, and has been repeatedly denied due

¶OUT to inadequate staffing. In its place, NASA Mission Assurance conducts a final "informal" review. Adjusting government inspection tasks is constrained by institutional dogma that the status quo is based on strong engineering logic, and should need no adjustment. This mindset inhibits the ability of Quality Assurance to respond to an aging system, changing workforce dynamics, and improvement initiatives.

TO CALL A TIME OUT

¶A TIME OUT may be called with or without this card Ref: FPP E-02_18, Time-Out Policy The Quality Planning Requirements Document, which defines inspection requirements, was well formulated but is not routinely reviewed. Indeed, NASA seems reluctant to add or subtract government inspections, particularly at Kennedy. Additions and subtractions are rare, and generally occur only as a response to obvious problems. For instance, NASA augmented wiring inspections after STS-93 in 1999, when a short circuit shut down two of Columbiaʼs Main Engine Controllers. Interviews confirmed that the current Requirements Document lacks numerous critical items, but conversely demands redundant and unnecessary inspections.

¶The NASA/United Space Alliance Quality Assurance processes at Kennedy are not fully integrated with each other, with Safety, Health, and Independent Assessment, or with Engineering Surveillance Programs. Individually, each plays a vital role in the control and assessment of the Shuttle as it comes together in the Orbiter Processing Facility and Vehicle Assembly Building. Were they to be carefully integrated, these programs could attain a nearly comprehensive quality control process. Marshall has a similar challenge. It

¶Figure 10.4-3. NASA Safety Reporting System Form.

¶Inspection requirements are specified in the Quality Planning is responsible for managing several different Shuttle systems through contractors who maintain mostly proprietary databases, and therefore, integration is limited. The main engine program overcomes this challenge by being centrally organized under a single Mission Assurance Division Chief who reports to the Marshall Center Director. In contrast, Kennedy has a separate Mission Assurance office working directly for each program, a separate Safety, Health, and Independent Assessment office under the Center Director, and separate quality engineers under each program. Observing the effectiveness of Marshall, and other successful Mission Assurance programs (such as at Johnson Space Center), a solution may be the consolidation of the Kennedy Space Center Quality Assurance program under one Mission Assurance office, which would report to the Center Director.

219

¶While reports by the 1986 Rogers Commission, 2000 Shuttle Independent Assessment Team, and 2003 internal Kennedy Tiger Team all affirmed the need for a strong and independent Quality Assurance Program, Kennedyʼs Program has taken the opposite tack. Kennedyʼs Quality Assurance program discrepancy-tracking system is inadequate to nonexistent.

¶Robust as recently as three years ago, Kennedy no longer has a "closed loop" system in which discrepancies and their remedies circle back to the person who first noted the problem. Previous methods included the NASA Corrective Action Report, two-way memos, and other tools that helped ensure that a discrepancy would be addressed and corrected. The Kennedy Quality Program Manager cancelled these programs in favor of a contractor-run database called the Quality Control Assessment Tool. However, it does not demand a closed-loop or reply deadline, and suffers from limitations on effective data entry and retrieval.

¶Kennedy Quality Assurance management has recently focused its efforts on implementing the International Organization for Standardization (ISO) 9000/9001, a process-driven program originally intended for manufacturing plants. Board observations and interviews underscore areas where Kennedy has diverged from its Apollo-era reputation of setting the standard for quality. With the implementation of International Standardization, it could devolve further. While ISO 9000/ 9001 expresses strong principles, they are more applicable to manufacturing and repetitive-procedure industries, such as

¶HEX Stamps Recorded FY01 thru FY03 (October 1, 2000 – April 2, 2003)

¶Number of HEX Stamps

Part Incorrect Part defective installation contaminated identification or damaged or fabrication

¶Figure 10.4-4. Rejection, or "Hex" stamps issued from October 2000 through April 2003.

¶running a major airline, than to a research-and-development, non-operational flight test environment like that of the Space Shuttle. NASA technicians may perform a specific procedure only three or four times a year, in contrast with their airline counterparts, who perform procedures dozens of times each week. In NASAʼs own words regarding standardization, "ISO 9001 is not a management panacea, and is never a replacement for management taking responsibility for sound decision making." Indeed, many perceive International Standardization as emphasizing process over product.

¶Efforts by Kennedy Quality Assurance management to move its workforce towards a "hands-off, eyes-off" approach are unsettling. To use a term coined by the 2000 Shuttle Independent Assessment Team Report, "diving catches," or last-minute saves, continue to occur in maintenance and processing and pose serious hazards to Shuttle safety. More disturbingly, some proverbial balls are not caught until after flight. For example, documentation revealed instances where Shuttle components stamped "ground test only" were detected both before and after they had flown. Additionally, testimony and documentation submitted by witnesses revealed components that had flown "as is" without proper disposition by the Material Review Board prior to flight, which implies a growing acceptance of risk. Such incidents underscore the need to expand government inspections and surveillance, and highlight a lack of communication between NASA employees and contractors.

¶Another indication of continuing problems lies in an opinion voiced by many witnesses that is confirmed by Board tracking: Kennedy Quality Assurance management discourages inspectors from rejecting contractor work. Inspectors are told to cooperate with contractors to fix problems rather than rejecting the work and forcing contractors to resub- mit it. With a rejection, discrepancies become a matter of record; in this new process, discrepancies are not recorded or tracked. As a result, discrepancies are currently not being tracked in any easily accessible database.

¶Of the 141,127 inspections subject to rejection from October 2000 through March 2003, only 20 rejections, or "hexes," were recorded, resulting in a statistically improbable discrepancy rate of .014 percent (see Figure 10.4-4). In interviews, technicians and inspectors alike confirmed the dubiousness of this rate. NASAʼs published rejection rate therefore indicates either inadequate documentation or an underused system. Testimony further revealed incidents of quality assurance inspectors being played against each other to accept work that had originally been refused.

¶Findings:

¶F10.4-1 Shuttle System industrial safety programs are in good health. F10.4-2 The Quality Planning Requirements Document,

¶Part FOD Open Incorrect which defines inspection conditions, was well work steps MIP

¶HEX stamps categories

formulated. However, there is no requirement that it be routinely reviewed.

¶F10.4-3 Kennedy Space Centerʼs current government mandatory inspection process is both inadequate and difficult to expand, which inhibits the ability

220

of Quality Assurance to process improvement initiatives.

¶F10.4-4 Kennedyʼs quality assurance system encourages inspectors to allow incorrect work to be corrected without being labeled "rejected." These opportunities hide "rejections," making it impossible to determine how often and on what items frequent rejections and errors occur.

¶Observations:

¶O10.4-1 Perform an independently led, bottom-up review of the Kennedy Space Center Quality Planning Requirements Document to address the entire quality assurance program and its administration. This review should include development of a responsive system to add or delete government mandatory inspections.

¶O10.4-2 Kennedy Space Centerʼs Quality Assurance programs should be consolidated under one Mission Assurance office, which reports to the Center Director.

¶O10.4-3 Kennedy Space Center quality assurance management must work with NASA and perhaps the Department of Defense to develop training programs for its personnel.

¶O10.4-4 Kennedy Space Center should examine which areas of International Organization for Standardization 9000/9001 truly apply to a 20-year- old research and development system like the Space Shuttle.