Reports that Matter

Columbia Accident Investigation Board Report, Volume I

ISS/SSP

Section 77 of 106

ISS/SSP

ISS Program Space Shuttle JSC Organization Manager Program Managers Manager

Shuttle Element Managers Endorse

Space Shuttle

S & MA Manager

United Space Alliance Vice President SQ & MA

quarters and decentralized execution of safety programs at the enterprise, program, and project levels. Headquarters dictates what must be done, not how it should be done. The operational premise that logically follows is that safety is the responsibility of program and project managers. Managers are subsequently given flexibility to organize safety efforts as they see fit, while NASA Headquarters is charged with maintaining oversight through independent surveillance and assessment.28 NASA policy dictates that safety programs should be placed high enough in the organization, and be vested with enough authority and seniority, to "maintain independence." Signals of potential danger, anomalies, and critical information should, in principle, surface in the hazard identification process and be tracked with risk assessments supported by engineering analyses. In reality, such a process demands a more independent status than NASA has ever been willing to give its safety organizations, despite the recommendations of numerous outside experts over nearly two decades, including the Rogers Commission (1986), General Accounting Office (1990), and the Shuttle Independent Assessment Team (2000).

Safety Organization Structure

Center safety organizations that support the Shuttle Program are tailored to the missions they perform. Johnson and

Issue: Same Individual,4 See also comments by Robert F. Thompson, Columbia Accident Shuttle Development," June 8, 1986, p. 1-7. Investigation Board Public Hearing, April 23, 2003, in Appendix G. 16 The 1971 cost-per-flight estimate was $7.7 million; $140.5 million dollars "Report of the Advisory Committee on the Future of the U.S. Space Program," December 1990, p. 2. 212 Report Volume I August 2003 JSC Form 564 (March 24, 2003). roles that cross Center, Program and NASA Administrator Headquarters responsibilies

Result: Failure of checks and balances

(Safety Advisor) Code Q Safety and Mission Assurance AA

Code Q MMT Letter

Space Shuttle SR & QA Manager

or l Input JSC SR & QA

Director

& QA Director

Independent Space Shuttle Division Chief Assessment Office

Space Shuttle

Organization

Managers d Task Agreements

Responsibility

Policy/Advice Figure 7.4-1. Independent safety checks and balance failure.

186

Marshall Safety and Mission Assurance organizations are organized similarly. In contrast, Kennedy has decentralized its Safety and Mission Assurance components and assigned them to the Shuttle Processing Directorate. This management change renders Kennedyʼs Safety and Mission Assurance structure even more dependent on the Shuttle Program, which reduces effective oversight.

At Johnson, safety programs are centralized under a Director who oversees five divisions and an Independent Assessment Office. Each division has clearly-defined roles and responsibilities, with the exception of the Space Shuttle Division Chief, whose job description does not reflect the full scope of authority and responsibility ostensibly vested in the position. Yet the Space Shuttle Division Chief is empowered to represent the Center, the Shuttle Program, and NASA Headquarters Safety and Mission Assurance at critical junctures in the safety process. The position therefore represents a critical node in NASAʼs Safety and Mission Assurance architecture that seems to the Board to be plagued by conflict of interest. It is a single point of failure without any checks or balances.

Johnson also has a Shuttle Program Safety and Mission Assurance Manager who oversees United Space Allianceʼs safety organization. The Shuttle Program further receives program safety support from the Centerʼs Safety, Reliability, and Quality Assurance Space Shuttle Division. Johnsonʼs Space Shuttle Division Chief has the additional role of Shuttle Program Safety, Reliability, and Quality Assurance Manager (see Figure 7.4-1). Over the years, this dual designation has resulted in a general acceptance of the fact that the Johnson Space Shuttle Division Chief performs duties on both the Centerʼs and Programʼs behalf. The detached nature of the support provided by the Space Shuttle Division Chief, and the wide band of the positionʼs responsibilities throughout multiple layers of NASAʼs hierarchy, confuses lines of authority, responsibility, and accountability in a manner that almost defies explanation.

A March 2001 NASA Office of Inspector General Audit Report on Space Shuttle Program Management Safety Observations made the same point:

The job descriptions and responsibilities of the Space

Shuttle Program Manager and Chief, Johnson Safety

Office Space Shuttle Division, are nearly identical with each official reporting to a different manager. This overlap in responsibilities conflicts with the SFOC [Space

Flight Operations Contract] and NSTS 07700, which requires the Chief, Johnson Safety Office Space Shuttle

Division, to provide matrixed personnel support to the

Space Shuttle Program Safety Manager in fulfilling requirements applicable to the safety, reliability, and quality assurance aspects of the Space Shuttle Program.

The fact that Headquarters, Center, and Program functions are rolled-up into one position is an example of how a carefully designed oversight process has been circumvented and made susceptible to conflicts of interest. This organizational construct is unnecessarily bureaucratic and defeats NASAʼs stated objective of providing an independent safety function. A similar argument can be made about the placement of quality assurance in the Shuttle Processing Divisions at Kennedy, which increases the risk that quality assurance personnel will become too "familiar" with programs they are charged to oversee, which hinders oversight and judgment.

The Board believes that although the Space Shuttle Program has effective safety practices at the "shop floor" level, its operational and systems safety program is flawed by its dependence on the Shuttle Program. Hindered by a cumbersome organizational structure, chronic understaffing, and poor management principles, the safety apparatus is not currently capable of fulfilling its mission. An independent safety structure would provide the Shuttle Program a more effective operational safety process. Crucial components of this structure include a comprehensive integration of safety across all the Shuttle programs and elements, and a more independent system of checks and balances.

Safety Process

In response to the Rogers Commission Report, NASA established what is now known as the Office of Safety and Mission Assurance at Headquarters to independently monitor safety and ensure communication and accountability agency-wide. The Office of Safety and Mission Assurance monitors unusual events like "out of family" anomalies and establishes agency-wide Safety and Mission Assurance policy. (An out-of-family event is an operation or performance outside the expected performance range for a given parameter or which has not previously been experienced.) The Office of Safety and Mission Assurance also screens the Shuttle Programʼs Flight Readiness Process and signs the Certificate of Flight Readiness. The Shuttle Program Manager, in turn, is responsible for overall Shuttle safety and is supported by a one-person safety staff.

The Shuttle Program has been permitted to organize its safety program as it sees fit, which has resulted in a lack of standardized structure throughout NASAʼs various Centers, enterprises, programs, and projects. The level of funding a program is granted impacts how much safety the Program can "buy" from a Centerʼs safety organization. In turn, Safety and Mission Assurance organizations struggle to antici- pate program requirements and guarantee adequate support for the many programs for which they are responsible.

It is the Boardʼs view, shared by previous assessments, that the current safety system structure leaves the Office of Safety and Mission Assurance ill-equipped to hold a strong and central role in integrating safety functions. NASA Headquarters has not effectively integrated safety efforts across its culturally and technically distinct Centers. In addition, the practice of "buying" safety services establishes a relationship in which programs sustain the very livelihoods of the safety experts hired to oversee them. These idiosyncra- sies of structure and funding preclude the safety organization from effectively providing independent safety analysis.

The commit-to-flight review process, as described in Chapters 2 and 6, consists of program reviews and readiness polls that are structured to allow NASAʼs senior leaders to assess mission readiness. In like fashion, safety organizations affiliated with various projects, programs, and Centers at NASA, conduct a Pre-launch Assessment Review of safety preparations and mission concerns. The Shuttle Program does not officially sanction the Pre-launch Assessment Review, which updates the Associate Administrator for Safety and Mission Assurance on safety concerns during the Flight Readiness Review/Certification of Flight Readiness process.

187

The Johnson Space Shuttle Safety, Reliability, and Quality Assurance Division Chief orchestrates this review on behalf of Headquarters. Note that this division chief also advises the Shuttle Program Manager of Safety. Because it lacks independent analytical rigor, the Pre-launch Assessment Review is only marginally effective. In this arrangement, the Johnson Shuttle Safety, Reliability, and Quality Assurance Division Chief is expected to render an independent assessment of his own activities. Therefore, the Board is concerned that the Pre-Launch Assessment Review is not an effective check and balance in the Flight Readiness Review.

Given that the entire Safety and Mission Assurance organization depends on the Shuttle Program for resources and simultaneously lacks the independent ability to conduct detailed analyses, cost and schedule pressures can easily and unintentionally influence safety deliberations. Structure and process places Shuttle safety programs in the unenviable position of having to choose between rubber-stamping engineering analyses, technical efforts, and Shuttle program decisions, or trying to carry the day during a committee meeting in which the other side almost always has more information and analytic capability.

NASA Barriers to Communication: Integration, Information Systems, and Databases

By their very nature, high-risk technologies are exceptionally difficult to manage. Complex and intricate, they consist of numerous interrelated parts. Standing alone, components may function adequately, and failure modes may be anticipated. Yet when components are integrated into a total system and work in concert, unanticipated interactions can occur that can lead to catastrophic outcomes.29 The risks inherent in these technical systems are heightened when they are produced and operated by complex organizations that can also break down in unanticipated ways. The Shuttle Program is such an organization. All of these factors make effective communication – between individuals and between programs – absolutely critical. However, the structure and complexity of the Shuttle Program hinders communication.

The Shuttle Program consists of government and contract personnel who cover an array of scientific and technical disciplines and are affiliated with various dispersed space, research, and test centers. NASA derives its organizational complexity from its origins as much as its widely varied missions. NASA Centers naturally evolved with different points of focus, a "divergence" that the Rogers Commission found evident in the propensity of Marshall personnel to resolve problems without including program managers outside their Center – especially managers at Johnson, to whom they officially reported (see Chapter 5).

Despite periodic attempts to emphasize safety, NASAʼs frequent reorganizations in the drive to become more efficient reduced the budget for safety, sending employees conflicting messages and creating conditions more conducive to the development of a conventional bureaucracy than to the maintenance of a safety-conscious research-and-development organization. Over time, a pattern of ineffective communication has resulted, leaving risks improperly defined, problems unreported, and concerns unexpressed.30 The question is, why?

The transition to the Space Flight Operations Contract – and the effects it initiated – provides part of the answer. In the Space Flight Operations Contract, NASA encountered a completely new set of structural constraints that hindered effective communication. New organizational and contractual requirements demanded an even more complex system of shared management reviews, reporting relationships, safety oversight and insight, and program information development, dissemination, and tracking.

The Shuttle Independent Assessment Teamʼs report documented these changes, noting that "the size and complexity of the Shuttle system and of the NASA/contractor relationships place extreme importance on understanding, communication, and information handling."31 Among other findings, the Shuttle Independent Assessment Team observed that:

  • The current Shuttle program culture is too insular
  • There is a potential for conflicts between contractual and programmatic goals
  • There are deficiencies in problem and waiver-tracking systems
  • The exchange of communication across the Shuttle program hierarchy is structurally limited, both upward and downward.32

The Board believes that deficiencies in communication, including those spelled out by the Shuttle Independent Assessment Team, were a foundation for the Columbia accident. These deficiencies are byproducts of a cumbersome, bureaucratic, and highly complex Shuttle Program structure and the absence of authority in two key program areas that are responsible for integrating information across all programs and elements in the Shuttle program.

Integration Structures

NASA did not adequately prepare for the consequences of adding organizational structure and process complexity in the transition to the Space Flight Operations Contract. The agencyʼs lack of a centralized clearinghouse for integration and safety further hindered safe operations. In the Boardʼs opinion, the Shuttle Integration and Shuttle Safety, Reliability, and Quality Assurance Offices do not fully integrate information on behalf of the Shuttle Program. This is due, in part, to an irregular division of responsibilities between the Integration Office and the Orbiter Vehicle Engineering Office and the absence of a truly independent safety organization.

Within the Shuttle Program, the Orbiter Office handles many key integration tasks, even though the Integration Office appears to be the more logical office to conduct them; the Orbiter Office does not actively participate in the Integration Control Board; and Orbiter Office managers are actually ranked above their Integration Office counterparts. These uncoordinated roles result in conflicting and erroneous information, and support the perception that the Orbiter Office is isolated from the Integration Office and has its own priorities.

188

The Shuttle Programʼs structure and process for Safety and Mission Assurance activities further confuse authority and responsibility by giving the Programʼs Safety and Mission Assurance Manager technical oversight of the safety aspects of the Space Flight Operations Contract, while simultaneously making the Johnson Space Shuttle Division Chief responsible for advising the Program on safety performance. As a result, no one office or person in Program management is responsible for developing an integrated risk assessment above the sub-system level that would provide a comprehensive picture of total program risks. The net effect is that many Shuttle Program safety, quality, and mission assurance roles are never clearly defined.

Safety Information Systems

Numerous reviews and independent assessments have noted that NASAʼs safety system does not effectively manage risk. In particular, these reviews have observed that the processes in which NASA tracks and attempts to mitigate the risks posed by components on its Critical Items List is flawed. The Post Challenger Evaluation of Space Shuttle Risk Assessment and Management Report (1988) concluded that:

The committee views NASA critical items list (CIL) waiver decision-making process as being subjective, with little in the way of formal and consistent criteria for approval or rejection of waivers. Waiver decisions appear to be driven almost exclusively by the design based Failure Mode Effects Analysis (FMEA)/CIL retention rationale, rather than being based on an integrated assessment of all inputs to risk management.

The retention rationales appear biased toward proving that the design is "safe," sometimes ignoring significant evidence to the contrary.

The report continues, "… the Committee has not found an independent, detailed analysis or assessment of the CIL retention rationale which considers all inputs to the risk assessment process."33 Ten years later, the Shuttle Independent Assessment Team reported "Risk Management process erosion created by the desire to reduce costs …" 34 The Shuttle Independent Assessment Team argued strongly that NASA Safety and Mission Assurance should be restored to its previous role of an independent oversight body, and Safety and Mission Assurance not be simply a "safety auditor."

The Board found similar problems with integrated hazard analyses of debris strikes on the Orbiter. In addition, the information systems supporting the Shuttle – intended to be tools for decision-making – are extremely cumbersome and e following addresses the hazard tracking tools and major tabases in the Shuttle Program that promote risk manage- nt. • Hazard Analysis: A fundamental element of system safety is managing and controlling hazards. NASAʼs only guidance on hazard analysis is outlined in the

Methodology for Conduct of Space Shuttle Program

Hazard Analysis, which merely lists tools available.35

Therefore, it is not surprising that hazard analysis processes are applied inconsistently across systems, subsystems, assemblies, and components.

United Space Alliance, which is responsible for both

Orbiter integration and Shuttle Safety Reliability and

Quality Assurance, delegates hazard analysis to Boeing. However, as of 2001, the Shuttle Program no longer requires Boeing to conduct integrated hazard analyses. Instead, Boeing now performs hazard analysis only at the sub-system level. In other words, Boeing analyzes hazards to components and elements, but is not required to consider the Shuttle as a whole. Since the current Failure Mode Effects Analysis/Critical Item

List process is designed for bottom-up analysis at the component level, it cannot effectively support the kind of "top-down" hazard analysis that is needed to inform managers on risk trends and identify potentially harmful interactions between systems.

The Critical Item List (CIL) tracks 5,396 individual

Shuttle hazards, of which 4,222 are termed "Critical-