Celebrating Apollo's 50th Anniversary: The Oldest Code on GitHub

Second of a three-part series to commemorate the 50th anniversary of the Moon landings.

Retrocomputing enthusiasts recently uploaded Apollo Guidance Computer (AGC) source code for various Apollo missions to GitHub. There is even a Virtual AGC that can run this code.⁷ In all likelihood, it is the oldest active code on GitHub. Remarkably, its development began over half a century ago in the ashes of Mariner 1, a Venus probe destroyed shortly after launch because of a bug in its guidance software.^15,35 The prevailing explanation was that in the transcription of hand-written guidance equations into a software specification for the contractor, TRW, an overbar to indicate the use of average rather than instantaneous velocity went missing, and along with it an $18M probe ($152M in 2019 dollars) and a regrettable ration of American prestige. How would MIT and NASA avoid similar mistakes developing software for the AGC?

This is the second of three articles about the AGC. In part 1, we described the hardware.²⁰ Here, in part 2, we focus on MIT's effort to develop the software.^4,12,36 As in part 1, the scientific computing community will recognize some familiar themes such as the benefits and challenges of co-design, the importance of sufficient testing resources, the role and impact of software process improvements and more.

Extreme co-design

Initially, about all that was known with any certainty was that a digital computer would be the centerpiece of a complex collection of GN&C subsystems. A year would pass before NASA selected the Lunar Orbit Rendezvous³⁷ (LOR) mission plan involving two separate and substantially different vehicles each with its own AGC. AGC software would control everything. Even so-called manually controlled inputs would first pass through AGC software before affecting the relevant hardware making the Apollo spacecraft the first all-digital fly-by-wire⁴⁹ vehicles ever created.

A challenge in developing the software was that all the GN&C subsystems pictured above were under development simultaneously right along with the software itself. Their interfaces, performance characteristics, size, weight and position within the spacecraft, all of which affect key parameters in the guidance equations, were constantly evolving. Even techniques to manage the software effort were under development and evolving with the software. Eventually, NASA would pressure MIT to adopt techniques pioneered by IBM to help manage large software development projects.

Today, we would call all of this simultaneous development activity co-design,^8,9,23 and it has a lot of advantages. But, in the 1960s when there were no DevOps³⁸ best practices and collaborative tools such as GitLab, Jenkins, Confluence, Kanban and WebEx, or even email, it presented a massive coordination and management challenge.

Evolving requirements, versions and flight rope releases

NASA had established a need for the machine and had determined its general tasks, and MIT received a contract based on only a short, very general requirements statement. Requirements started changing immediately and continued to change throughout the program.³⁹

Midway through development, the AGC was redesigned to support more memory and an expanded instruction set. The original and redesigned systems were different enough that each required separate software development teams, a situation that only worsened already strained resources. The redesigned AGC had only 2K words of erasable core and 36K words of fixed or rope core memory. All software and data had to fit into this combined 76KB of memory.

The essential step for software developers was to produce a flight program or rope and release it to Raytheon for rope core manufacture approximately 4 months prior to launch: 2 months to manufacture the ropes followed by 2 months for installation in the spacecraft, checkout, integrated system testing, crew rehearsals and final erasable memory load.

MIT needed to deliver flight programs for ~30 Apollo flights (crewed and uncrewed), many with unique guidance requirements, planned between 1966 and 1972.⁴⁰ The lead engineer in coordinating and approving a completed flight program was called a rope mother and would also name the release. Early on, the names were fairly creative, including ECLIPSE, SUNRISE, RETREAD and AURORA. Eventually, NASA put a stop to this, and only the names COLOSSUS and LUMINARY together with their revision numbers would be used to identify CM and LM flight programs, respectively.²⁴ LUMINARY 1A is the revision used in the first lunar landing of Apollo 11.

The AGC software stack

Each flight program involved a combination of common utilities and mission-specific space guidance subroutines. Mission-specific components required significant analysis and development time. Early development activity, 1961–1965, focused on infrastructural software.

Program Name	Purpose	Size (AGC words)
Executive²⁵	Priority-driven large/long-running process manager	~350
Waitlist²⁶	Time-sequenced small/short-running process manager	~300
Down-Telemetry²⁹	Transmit system data to ground	~200
Restart^30,31,32	Error recovery and restart protection	~1225
Interpreter²⁷	Space guidance domain-specific programming language interpreter	~2200
DSKY I/O²⁸	Cockpit displays and keypad	~3500
Combined Total	22% of fixed memory	~7775

These programs constituted what we might call today the Apollo guidance software stack. All were implemented in assembly language. By 1965, most of this code had been written and fully tested and changed little with each new flight program. All higher level space guidance routines were implemented primarily in the Interpreter language¹³ but also by using some of these lower-level pieces.

An example of a space guidance subroutine is computing the relative positions of Earth, Sun and Moon at any moment. After evaluating options³⁴ in MAC⁵⁰ or Fortran on mainframe systems, developers settled on an approach using 8th-degree polynomial fits to time-varying positional data predicted from mainframe solution of the 3-body (Earth, Sun, Moon) problem. Eight double-precision X, Y and Z polynomial coefficients, 48 words of data, fitting a 2-week period of Moon position data would then be stored in fixed memory. Another example is a list of stars⁴⁷ and spatial positions used with the Apollo space sextant³³ requiring 112 words.⁴¹ This data and code would be among the 76 kilobytes of a flight program hand woven into rope core in the months before launch. For time-sensitive data, multiple ropes for different launch windows would be manufactured as contingencies. A 1962 memo¹⁹ lists 45 major software analysis efforts then underway for various aspects of planned Apollo missions.

The AGC had an app for that

Flying to the moon and returning safely involved long periods of boredom punctuated by moments of extreme peril. A mission was divided into phases by velocity change maneuvers or burns of the main engines¹. A complete mission involved around 11 burns. For each maneuver there was a corresponding major mode program, to handle it. For every phase of the mission, the AGC had an app for that.⁴⁵

By far the most critical sequence of maneuvers occurred during lunar landing. It was divided into four phases (pictured below left) depending on the balance of automated and manual control the astronauts required: Powered Descent (major mode P63), Approach (P64), Terminal Descent (P66) and Touchdown (P68).⁵ Examples of other major mode programs were Trans Lunar Injection (P15), Return To Earth (P37) and Ballistic Re-entry (P66).

Development of a major mode program began with an analysis of the relevant equations of motion and an assessment of available computational approaches to affect the desired velocity change (magnitude and direction) subject to numerous considerations including zero gravity fuel slosh; changing center of mass due to fuel consumption; main engine performance characteristics; sensor drift and deadbands (e.g. IMU gimbal lock); optimizing use of RCS propellants; contingencies for failed (on or off) RCS thrusters; the Moon's lump gravity field;^2,3 and precision timing to coordinate a planetary ballet of Earth, Moon, Sun and multiple spacecraft and the lines of sight of communications between them and mission control.

Performance portability and the digital autopilot (DAP)

Digital Autopilot (DAP) software was developed based on Kalman Filtering.⁴² The computation is decomposed into a prediction phase where an idealized model of the spacecraft is used to estimate the current state. In the second phase, noisy direct measurement of system state from spacecraft sensors is compared with the predicted state to produce control decisions.

A key challenge was ensuring that a single implementation of DAP software would provide effective control given a wide variety of spacecraft configurations and operating scenarios. Doing so presented what we would call a performance portability problem.¹⁰ Software developers made DAP execution configurable through a number of parameters. Prior to a burn, astronauts would follow a checklist setting a number of switches and entering data on the DSKY to set parameters for DAP execution during the burn.⁴⁶

Software developers were given a budget of 10% of rope core memory and 20-30% of full computational load (3-4.5 kFLOPS) in which the DAP would have to operate. It would take four developers three years and 2,000 words of rope core to develop the LM DAP software alone. A key optimization realized late in development was that a change in coordinates used in the computations from body axes to jet axes reduced complexity and increased performance.^6,11

The picture above, right shows the nonlinear switching logic used by the Kalman filtering algorithm to control RCS jet firings in coasting flight. With a change of a dial on the control panel, astronauts could adjust the filter from coarse to fine control.

AGC software testing: 60% of the whole effort

Five different levels of testing were developed.

An all-software simulator (also known as the all-digital simulator) for the AGC, which included simulation of the environment.⁵¹ A key concern was whether the environment faithfully represented the behavior of the real GN&C hardware and spacecraft including such issues as engine performance, fuel slosh and even structural responses of the spacecraft under torques and loads imposed by engine gimbaling and thruster firings. The all-digital simulation was implemented in MAC and Fortran on MIT mainframes and ran at 10% of real-time speed. All-digital simulations were bit-for-bit repeatable.
A hybrid simulator using a real AGC together with a rope core simulator and a combination of digital and analog simulation of various GN&C components, the latter requiring two massive machine rooms in the second and third stories of the test facility. Hybrid simulations were not repeatable.
System test labs using a real AGC and real GN&C sensor systems such as the IMU, radar and optics. These were used primarily to test AGC hardware and software response to noisy inputs from these components.
Crew rehearsals with a real AGC in the actual spacecraft exercising some of the actual GN&C subsystems. As a practical matter, such tests of course could not include engine or thruster firings but did exercise other system components.
Actual flight tests of the fully integrated GN&C system both in uncrewed and crewed flights. Early missions included several objectives designed specifically to test AGC software.

In the data pictured here, data from actual flight tests of the LM descent engine is compared with the all-digital simulation. The inner gimbal angle data agree reasonably well (left). However, a clear bug is revealed (middle) in failure to faithfully model the outer gimbal angle because of missing structural dynamics modeling, which was eventually corrected (right).

The need for formal validation rose with the size of the software. Programs of 2,000 words took between 50 and 100 test runs to be fully debugged, and full-size flight program took from 1,000 to 1,200 runs.¹⁴

The all-digital simulation of the AGC would eventually require MIT to purchase one Honeywell 800, two Honeywell-1800s and two IBM 360/75 peaking at about 4,500 CPU-hours/month (equiv. H-1800 CPU-hour) testing solely for the all-digital test simulator. These hardware testing resources together with simulator software development and test operators comprised nearly 60% of the entire software development budget.

Putting the software effort in context

The whole Apollo GN&C system cost about $600M over ten years¹⁸. The software alone was about 10% of that or about $60M¹⁷, the majority of it occurring over the last five years. This equates to $100M/year in 2019 dollars.

Before the first lunar landing, more than 1400 person-years of software engineering effort had been expended, with a peak level of effort of 350 engineers reached in 1968.^17,52

A 1972 master's thesis¹⁷ breaks down software costs by category shown below, left. Factoring out the Computer category used almost exclusively for testing, the adjusted, relative costs of the software development alone are shown below, right. To help keep documentation costs down, there was even a computer-automated documentation system developed.⁴³

AGC software development costs by category: left includes test hardware costs and right with hardware costs factored out.

In the early stages, there were no "programmers." Instead, engineers and scientists learned the techniques of programming. It was believed that competent engineers could learn programming more easily than programmers could learn engineering.³⁶

We can thank Margaret Hamilton, who received the Presidential Medal of Freedom for her work on the on AGC^16,22, for being the first to champion software engineering²¹ as a discipline unto itself "to bring the software [effort] legitimacy so that it and those building it would be given due respect." Hamilton was the only woman working on AGC software and ultimately became a rope mother for the LM fight program LUMINARY.

Throughout much of the Apollo effort, MIT experienced difficulty in estimating the time and effort requirements to design, test and verify successive mission programs.³⁶

No one doubted the quality of the software eventually produced by MIT. It was the process used in software development that caused great concern. Five lessons were identified: (1) up-to-date documentation is crucial, (2) verification must proceed through several levels, (3) requirements must be clearly defined and carefully managed, (4) good development plans should be created and executed, and (5) more programmers do not mean faster development.¹⁴

The Russian program achieved all of its early successes using ground-based computers for guidance. This approach is possible for Earth orbital flights and a single vehicle. Providing sufficiently accurate and timely guidance for multiple vehicles or lunar missions including soft landing and return to Earth eventually forced the Russians to begin their own digital, on-board computer development. In August 1969, the uncrewed Russian probe Zond-7⁴⁸ guided by an Argon-11S⁴⁴ digital computer completed the first fully successful Russian circumlunar mission.

Part 1 | Part 2 | Part 3

Author bio

Mark C. Miller is a computer scientist supporting the WSC program at LLNL since 1995. Among other things, he contributes to VisIt, Silo, HDF5 and IDEAS-ECP.

Celebrating Apollo's 50th Anniversary: The Oldest Code on GitHub

Extreme co-design

Evolving requirements, versions and flight rope releases

The AGC software stack

The AGC had an app for that

Performance portability and the digital autopilot (DAP)

AGC software testing: 60% of the whole effort

Putting the software effort in context

Author bio

References

Comment

More on High-Performance Computing (HPC) and Performance Portability

I/O Sleuthing: Digging into Storage Performance

BSSw Fellowship

Julia's Value Proposition for Better Scientific Software

Deep Dive

Containers for Deploying Workflow Systems and Application Codes

BSSw Fellowship

Rethinking Software Variants

Deep Dive

Improving Application Performance by Optimizing I/O

BSSw Fellowship and How To