software-engineering-notes

Introduction

"Those who cannot remember the past are condemned to repeat it." -- George Santayana

I began my career in aerospace engineering in the fall of 2019 at a small company in Houston, Texas. There, I came under the mentorship of one of the most striking individuals I have ever known in my life, who I will call Harry. Harry is simultaneously stunningly brilliant, and maddeningly eccentric. He is an expert in orbital mechanics, analytical methods, and a language lawyer¹[An expert of a programming language]] who is extremely well-versed in its specification.] in C+. He has forgotten more about C than most people will ever learn; he was fond of using every little trick he's found whenever the opportunity arises, whether it's called for or not, to the chagrin of the rest of us. Most importantly, however, Harry was a mentor for many people at my company, myself included. His eccentricity was more charming than annoying, and his expertise was so highly valued that he had to split his time among many projects simultaneously to provide technical guidance. I was always jealous of him being "stolen" from us for his other projects. What made Harry a great mentor? I could talk to him briefly about whatever I was working on, describing where I was stuck. Through these dialogues, we would determine the problem underlying surface issue, and I learned something new. Over the brief time I knew him, we built a strong rapport, and he left an indelible mark in my memory. Harry holds the qualities of a good engineer: curious, inquisitive, skeptical, cooperative, communicative, always learning, and a great teacher.

I aspired to become Harry for a while; I eventually realized he truly was one of a kind, and that I am on my own journey. Why am I telling you about Harry? Because I am afraid that he is a dying breed of engineer. With the loss of people like Harry, more engineering mistakes will happen, causing more costly accidents, injuries, and death. I am afraid of what is happening to our industry. We have decided to take more than six decades of lessons learned about our profession and throw them away. It is my goal to teach you what good engineering needs. I will define the attitude required, and the skills you must hone to be a good engineer. It is my hope that you will take pride in your work, and constantly better yourself.

I will discuss the two NATO conferences on Software Engineering from 1968 and 1969, Turing's "Computing Machinery and Intelligence", Peter Naur's works, and Fred A. Brooks' "The Mythical Man Month." I will also recount my own experiences as a software engineer, and what I learned as a nuclear operator in the United States Navy. Additionally, I will look into works on engineering as varied as Vitruvius' "Ten Books on Architecture" and Admiral Rickover's "Thoughts on Man's Purpose in Life...and Other Matters". There is a surprising thread of philosophy woven into these works, touching upon Bertrand Russel, William James, Voltaire, and Karl Popper. There is timeless wisdom, practical advice, and inspirational calls for courage to be found in these works.

Please note that wherever I write "AI," you can comfortably substitute it with "LLM" unless otherwise noted. I do not deny that there are useful applications of these technologies, but it is not the purpose of this paper to state what those are or justify them. I will leave this introduction with a motivating quote:

It is important that we remember to distinguish between what it means to fail at a task and what it means to be mediocre. There is all the difference in the world between the life lived with dignity and style which ends up failing, and one which achieves power and glory yet is dull, unoriginal, unreflective, and mediocre. In a real sense, what matters is not so much whether we make a lot of money, hold a prestigious job, or whether we don’t; what matters is that we become people who seek out others with knowledge and enthusiasm—that we become people who can enjoy our own company. -- Hyman G. Rickover

We Face a New Crisis in Software Engineering

McClure: Confession is good for the soul —

d’Agapeyeff: — but bad for your career.

I must sound the alarm with the watchmen: AI is changing our industry for the worse. People are recklessly abandoning well-established practices and standards, and software is getting worse as a result. The cost of bad software is borne not only in a degraded user experience in services, applications and operating systems, but also in expensive downtime, materiel damage, and loss of life. There is growing evidence that use of AI is detrimental to the user's ability to think critically and to learn, while also increasing the stress on programmers. This matters because engineering is the application of understanding to reality; an engineer's job is to solve problems in systems safely, within the bounds of the constraints placed upon the systems, the time of the engineers and technicians involved, and the budgetary resources of the companies employing them. I have personally observed good engineers decide that they don't need to learn parts of their technology stack anymore. I believe this is a critical error in the estimation of the value of knowledge and understanding.

I think that a firm understanding of history is necessary to understand the nature and severity of our present challenges.

The First Software Crisis

This is not the first time that a crisis in software engineering has been observed. In 1968, the NATO Science Committee sponsored the Working Conference on Software Engineering. Many experts from academia and industry attended this conference: Douglas McIlroy of Bell Labs², Alan Perlis³[Epigrams in Programming.]]"], Edsger Dijkstra⁴, and Peter Naur⁵, among many others. The attendees realized that there was a crisis in Software Engineering: projects were running past their deadlines, over their budgets, and with fewer features than were promised. The crisis the attendees spoke of was decidedly different than what we face today; for them, the nature of the crisis was that they had not yet developed the techniques necessary to tame complexity. Perhaps it should come as no surprise that many of the problems they discussed have been perennial difficulties: the ability to accurately estimate how long it will take to produce software, lack of feedback from users, imprecise and changing specifications, and great difficulty in proving correctness. A frequently cited cause of schedule slip was a mixture of novelty (research) and production.

Fred Brooks echoes the problem of estimation in the opening of the eponymous essay from the book, "The Mythical Man Month", stating that estimating techniques are poorly developed, and engineers are too optimistic when they make them. He also notes that estimation techniques falsely equivocate effort with progress, "hiding the assumption that men and months are interchangeable." Brooks identifies that adding man-power to a late project only intensifies the problem, and he develops an argument about why this doesn't solve the problem: increasing communication costs. I will delve back into this later in this essay, and tie in a new reflection of Conway's Law, as it relates to AI and vibe coding. Brooks calls out complexity impeding testing as a major source of schedule slippage, and lays blame at the eternal optimism of programmers for naively thinking that testing will take much less time than it usually does.

Edsger Dijkstra observes in The Humble Programmer that the major cause of the crisis is that hardware has become so much more powerful, magnifying the repercussions for failure in proportion to the abilities conferred by the power of the computers: I believe this cause parallels our own.

NATO held a second conference in 1969, attended by many of the same people, as well as luminaries like Tony Hoare⁶. Although there was less focus on the software crisis, there was still significant discussion of problems: correctness, testing, debugging, and performance evaluation. One member noted the severity of the problem: "in OS/360 we have about 1000 errors each release." In a discussion on formal correctness, a debate emerged around the efficacy of testing, culminating in an exchange where Perlis advocates for rigorous testing. Dijkstra responds that "testing shows the presence, not the absence of bugs."

The original software crisis emerged as the scale of the problems that could be solved economically grew much faster than programming techniques could develop. Over the intervening decades, a tremendous amount of time and effort has been expended on improving programming techniques through language theory, psychology, and industry practices. We have seen the emergence of structural programming, procedural programming, object oriented programming, functional programming, data oriented design, optimization theory, modularization, unit testing, smoke testing, end to end testing, static analysis, dynamic analysis, extreme programming, and the agile methodologies, among many other developments. Fred Brooks quotes H. Coqui:

...the goal of applying Engineering principles to Software production in the 1970s was to increase the Quality, Testability, Stability, and Predictability of software products—not necessarily the efficiency of Software production. The driving force to use Software Engineering principles in software production was the fear of major accidents that might be caused by having uncontrollable artists responsible for the development of ever more complex systems. --

1. Coqui

Beneath all of these was the current of maintainability, comprehensibility, and correctness. That is not to say that the industry has not been rife with poor processes, dogmatism, undisciplined behaviors, and outright fraud; however, these traits were not celebrated.

Apollo Software notes

To be integrated into another section.

Apollo Software

Team rapport, discipline, RTOS, memory leak and hardware failure, display systems, modeling

Randell: It would seem to me that one of the main reasons for the undoubted success of the Apollo programming support system is its environment. It is part of a much larger effort, the total Apollo project, involving many other technological problems that had to be solved, and which could only be tackled a few at a time. Hence the support system has been done over many times; each time there has been a considerable change but it hasn’t been a revolution. What worry me are systems which don’t benefit from such a constraining environment, where software is the only technology that would be involved in a huge leap forward.

The Modern Software Crisis

The dissemination of knowledge is of obvious value — the massive dissemination of error-loaded software is frightening. -- Edsger Dijkstra

The era of vibe coding is here, and it has precipitated a new crisis. The AI companies are regularly releasing new agents and models, each more capable than the last, of producing code which has the appearance of functionality. Error rates for code produced have decreased through the application of new techniques by early adopters, but the fundamental problem of so-called hallucinations remains. A less anthropomorphized term for this phenomena is the generation of false information confidently presented as fact. However, this is not the worst problem I have with AI: by divorcing ourselves from the process of performing technical and creative work, we lose our ability to understand the the produced artifacts. This problem is amplified by the ability of AI to rapidly produce tens of thousands of lines of code per day. Don't take my word for it, just listen to Garry Tan:

Absolutely insane week for agentic engineering

37K LOC per day across 5 projects

Still speeding up -- Garry Tan

I'm trying to teach software engineers that their sacred cow of "LOC is bad" is probably wrong in agentic engineering

You won't believe that in another 5 years -- Garry Tan

For those who don't know, Garry Tan is the CEO of Y Combinator. You may have heard of them from such startups as Airbnb, Coinbase, DoorDash, Reddit, Stripe, Scale AI, Twitch, and Whatnot. They have had such illustrious members as Sam Altman of OpenAI and Peter Thiel of Palantir. Garry and other venture capitalists are heavily pushing the use of LLMs across all walks of life.

It is hard to believe that we have reverted to measuring productivity in lines of code. Measuring productivity by lines of code has a long history, and was widely abandoned because it was recognized to reward bad behaviors for the wrong reason. Adding more lines of code without evaluating the cost leads to increased complexity, and thus entropy. In the words of Fred Brooks, "program maintenance is an entropy-increasing process, and even its most skillful execution only delays the subsidence of the system into unfixable obsolescence ." Fraser noted in the 1968 NATO conference that a central problem in the software production process is how to identify the nature of progress and measure it: "program construction is not always a simple progression." It has always been very difficult to measure programmer productivity; similarly misguided metrics such as commits made or issues closed have been used to ill effect. A more reasonable model in recent times has been the consideration of good features shipped, but even this may be wanting. I think a discussion of the particular details of the evaluation of a programmer by quantifiable metrics that are easily observed by managers is somewhat beyond the scope of this paper. However, I hope that this paper will show managers what good qualitative properties to observe in their engineers.

You should note that Garry has deliberately called his activities "agentic engineering." This is an evolution of the term "prompt engineering", and a turn away from the easily and frequently maligned term of "vibe coding." Vibe coding itself was coined by Andrej Karpathy, the co-founder of OpenAI, in February 2025:

There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works. -- Andrej Karpathy

Proponents loudly exclaim that vibe coding and AI accelerates product development and multiplies the outputs of individuals, but what problems does this new era of programming with AI bring us?

Case Studies in Failures in Software Engineering and Their Causes

Software engineering has already had its fair share of disasters, even before the involvement of vibe coding. Take, for example, the delay in finding the ozone hole over Antarctica, the explosion of the Ariane 5 on its first flight, and the deaths caused by the Therac-25 radiation therapy machine.

Ozone Hole Discovery

Pawan Bhartia, Richard McPeters, and Gavin Schmidt write that despite the popular narrative that the discovery of the ozone hole over Antarctica came as a complete surprise to the scientific community, there were in fact people at NASA who already suspected its existence because of the data that they received from the Nimbus-7 satellite (launched in October of 1978) monitoring the atmosphere. Bhartia and McPeters state that the satellites measurements were flagged for unsuitability as data in the algorithm used to derive the ozone measurement in the atmosphere because they were too low -- outliers. These outliers were still reported with the data, just not used originally. In order to verify if the data itself was bad, they had to take in-situ measurements at the south pole. The measurements they took from the ground station at the South pole themselves turned out to be erroneous, leading to further delay. Had the ground software not been designed to mark the outliers as missing data, or in a positive sense, to alert the researchers that the majority of the data points were being rejected, action could have been taken sooner.

Contrary to the focus of the narrative that Bhartia presents, Zehr states that it "appeared that scientists' own actions and beliefs were not responsible for, or even part of, prior incorrect claims about ozone depletion. Rather, the focus was on incorrect models (treated as independent actors) and the unusualness of nature." Because the discovery was made so late, and in-situ investigation takes so much time, the discovery of the ozone hole was delayed by years. There was no programming error in the sense of a logical error, but a semantic error of a particular kind: the violation of assumptions. Bhartia and McPeters are right when they observe that other people knew there was something wrong with the data, but the fact remains that the discovery was delayed because the ground software did not ring the alarm bell. The next incident I will discuss is a semantic error of an entirely different type.

Ariane 5

The first flight of the Ariane 5, Ariane 5 Flight 501, ended in disaster in 1996, when it exploded not even a minute after taking off from the flight pad. The European Space Agency (ESA) and the Centre National d'Études Spatiales (CNES) commissioned an inquiry Board chaired by Jacques-Louis Lions to investigate the cause of the failure. The Board's investigation determined that 37 seconds after launch initiation there was a failure of the back-up Inertial Reference System (SRI) ⁷ followed immediately by the failure of the active Inertial Reference System. After that, the rocket's steering system pointed it off course very quickly, inducing sufficient force and damage to trigger its self-destruction mechanism. The report isolates the SRI units as the origin of the failure.

The Board worked backwards in time from the self-destruction mechanism triggering to determine the root cause:

• Both SRIs had experienced a software exception and ceased functioning.
• The software exceptions were caused by an illegal operation in the program⁸
• This error occurred in a part of the software which was no longer necessary after lift-off.
• The code was mistakenly retained, since it was reused from software for the Ariane 4 which required it, but was not a requirement for the Ariane 5.

The immediate reason that the error occurred was a parameter was much higher than was expected by the design of the code, since it was designed for the Ariane 4, which had different flight characteristics than the Ariane

1. What circumstances led to the inclusion of the code in the first place, or why the code was

executing after launch, or why the SRIs were designed to fail after the exception?

The Board thought that the requirement to shutdown the SRIs in the event of any exception was misguided, and considered that providing a best estimate of attitude information in spite of the exception would have been a better policy, instead of shutting down the otherwise healthy and operational SRIs which were critical to the functionality of the rocket. The requirement which caused the code to continue operating after lift-off was found to be brought forward with no basis from the prior design. Other elements of the design of the Ariane 5 obviated the need for the code to continue executing after lift-off; the decision to keep the code was founded on the view that "it was not wise to make changes in software which worked well on Ariane 4," but analysis found that assumptions in the code were violated by post lift-off conditions. The Board stated that "software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct." Insufficient testing was identified, and it was noted that early in the testing design, the SRIs were supposed to be tested in a simulation: this would have identified the faulty software immediately. The decision to test the SRIs was changed since the SRIs were considered to be fully qualified at the equipment level, and the simulation environment was not considered realistic or accurate enough. The Board considered this reasoning to technically valid, but that testing anyway would have achieved proper system integration. The SRI software was not fully analyzed in reviews, and the review process itself was a factor in the failure.

Nuseibeh further argued that risk management was another cause of the failure: "a risky decision was not reviewed as the project evolved... risk can change as requirements change, risk is more likely to increase than decrease." Further, Nuseibeh states that it is a myth that computers are more reliable or reduce risk compared to the systems they replace.

I believe the Ariane 5 disaster is representative of the fact that software can be valid and correct in one circumstance, but completely wrong in another. By being aware of the entire system, as it evolves, we can be more cognizant and prepared for the changing circumstances, and mitigate the risks involved, by applying analytical techniques. The Board's final recommendation is salient for this paper:

A more transparent organisation of the cooperation among the partners in the Ariane 5 programme must be considered. Close engineering cooperation, with clear cut authority and responsibility, is needed to achieve system coherence, with simple and clear interfaces between partners.

Ariane 5 Flight 501 cost $370 million dollars, but took no lives, unlike the Therac-25 accidents.

Therac-25

Most accidents are system accidents; that is, they stem from complex interactions between various components and activities. To attribute a single cause to an accident is usually a serious mistake. -- Nancy Leveson and Clark

1. Turner

The Therac-25 was a computer controlled radiation therapy machine manufactured by Atomic Energy of Canada Limited (AECL)⁹. From June 1985 to January 1987, a combination of factors, ranging from race conditions in code to unethical manufacturer behaviors, led directly to the needless suffering and deaths of people across six accidents, who were exposed to 100 times the intended dose of radiation. The precise nature of the software bugs is not of interest in this paper, but instead the circumstances which allowed them to cause harm. The seminal report on the matter from Nancy Leveson and Clark S. Turner is highly illuminating and disturbing, and I recommend readers review the story in full. The stories of the victims are tragic.

Victims of the Therac-25 Accidents

Katie Yarbrough, a 61-year-old woman was undergoing radiation therapy on June 3, 1985 at the Kennestone Regional Oncology Center, in Marietta, Georgia. When the technician turned on the Therac-25, Katie felt a burning sensation. Two weeks after the incident, Tim Still, the physicist at Kennestone noticed that the patient had a a burn on her back matching the front of her shoulder where she initially reported the burning sensation. Still estimated that she received "one or two doses of radiation in the 15,000 to 20,000 rad (radiation absorbed dose) range.¹⁰ The woman's breast had to be removed because of the severity of the radiation burns, and she lost the use of her shoulder and arm, leaving her in constant pain. The patient filed a lawsuit in the autumn of 1985 against Kennestone, AECL, and the hospital's Therac-25 support service organization. The accident was not reported to the FDA until the accidents in Tyler Texas, and the lawsuit was settled out of court. Katie Yarbrough died in 1990 in a car accident.

Frances Hill, A 40-year-old woman receiving treatment for cervical cancer at the Ontario Cancer Foundation in Hamilton, Ontario was the second recorded person to suffer an accident in the operation of the Therac-25 on July 26,

1. She complained that she felt like she had been

burned, describing it as an "electric tingling shock." She returned three days later for more treatment, and complained of burning and excessive swelling in the area targeted by the machine. Later, one of the AECL's own technicians estimated that Frances Hill had received 13,000 to 17,000 rads. Frances Hill died November 3, 1985, "of an extremely virulent cancer." An autopsy showed that Hill would have required a total hip replacement as a result of the radiation overexposure.

Janis Tilman was undergoing treatment at Yakima Valley Memorial Hospital, in Yakima, Washington, in December

1. After one of her treatments, she developed skin redness in a

striped pattern in the location she was being treated. Hospital staff were unable to produce a satisfactory for why this was the case, and contacted the AECL in January. The AECL technical support supervisor sent a written response in February to the director of radiation therapy in at Yakima denying that the Therac-25 could be responsible, and denied that there had been any other similar reports. The hospital staff reported the incident as having an unknown cause, but re-investigated it a year later after the second overdose at Yakima. They discovered that Janis Tilman was suffering tissue necrosis, and was in constant pain. Janis was given treatment, and survived with no further reports of complications.

Voyne Ray Cox, a 33-year-old man who had a tumor surgically removed from his back, was receiving follow-up treatment on March 21, 1986 at the East Texas Cancer Center (ETCC) in Tyler, Texas. While setting up the Therac-25 for the treatment, the operator noticed she had incorrectly typed the wrong letter for the treatment type, and quickly edited the entry to correct it. After continuing the procedure normally and starting it, the machine paused and gave an an enigmatic error message. The operator resumed the machine's operation, which paused again with the same error message. Cox left the room after realizing that something had gone horribly wrong, informing the operator that he had felt a burning sensation or an electric shock and had heard a buzzing sound from the equipment. A physician examined Cox, and found severe skin reddening over the treatment area, but thought that electric shock was the cause. Cox was discharged with instructions to return if any new developments arose. Simulations later indicated that Cox received a dose between 16,500 to 25,000 rads to the center of his upper back. In the weeks after the accident, Cox developed the classic symptoms of acute radiation syndrome. He was hospitalized for the radiation-induced myelitis of his spinal cord which paralyzed his left arm, both legs, left vocal cord, and left diaphragm, among other symptoms. Voyne Ray Cox died from acute radiation syndrome later that year.

Verdon Kidd, a 66-year-old man, was receiving treatment for skin cancer on his face at ETCC. On April 11, 1986, the same operator who set up the Therac-25 for Voyne Ray Cox made the same input error to the Therac-25's terminal, and performed the same editing technique to correct her mistake. When the machine paused and showed the same mysterious error from before, she heard the patient moaning over the intercom. When she entered the room to ask him what was wrong, he said that he felt fire on his face. Verdon Kidd died on May 1, 1986, in a coma with hyperpyrexia (a fever of 104° F or higher), having suffered from radiation damage to his brain.

Glen Dodd, a 65-year-old man was receiving treatment at Yakima valley Memorial Hospital on January 17,

1. The operator of the machine performed two

initial doses without issue, but the third application went wrong, pausing the machine. The operator resumed the treatment, and the machine paused again, and heard Glen over the intercom. Glen reported "feeling a burning sensation" in his chest, and later in the day developed a skin burn over his treatment area. The burn took on the striped pattern that Janis Tilman experienced previously. Glen Dodd died in April, 1987, from complications due to the overdose. A lawsuit filed by his survivors alleging that he suffered unnecessarily was settled out of court.

Causes of the Therac-25 Accidents

Accidents are seldom simple - they usually involve a complex web of interacting events with multiple contributing technical, human, and organizational factors. One of the serious mistakes that led to the multiple Therac-25 accidents was the tendency to believe that the cause of an accident had been determined (...) without adequate evidence to come to this conclusion and without looking at all possible contributing factors. Another mistake was the assumption that fixing a particular error (eliminating the current software bug) would prevent future accidents. There is always another software bug. -- Nancy Leveson and Clark S. Turner

Throughout the case history that developed during the operation of the Therac-25 between June 1985 and January 1987, AECL exhibited a pattern of denial and dishonesty. Despite being informed on multiple occasions, they repeatedly told inquiring hospital staff that it was impossible to operate the machine improperly, and that they were unaware of any other incidents. This pattern persisted until the second incident at ETCC when Verdon Kidd was fatally overexposed to radiation by the Therac-25, despite the lawsuit filed against them by Katie Yarbrough in November 1985. AECL's initial response to the reports from the Ontario Cancer Foundation was to propose that there were hardware issues in the design, and to release modifications to ameliorate that potential flaw, claiming that it represented an improvement of safety by five orders of magnitude. This claim led staff at Yakima Valley Memorial Hospital to believe that the machine could not be at fault, setting the stage for Glen Dodd's later suffering. The hospital staff at ETCC, the Ontario Cancer Foundation, and the Yakima Valley Memorial Hospital, as well as members of the United States Food And Drug Administration (FDA) and the Candian Bureau of Radiation and Medical Devices (BRMD), found that significant problems existed in the software of the Therac-25. AECL finally found that there were significant problems in the software after the fatal accidents of Verdon Kidd and Glen Dodd.

At least part of the code used by the Therac-25 was originally written for the Therac-6. This code was also reused for the Therac-20, but both the Therac-6 and Therac-20 featured hardware interlocks that prevented the kinds of accidents that the victims of Therac-26 suffered. AECL chose not to duplicate these hardware interlocks for the Therac-25, likely as a means to save money. The code was written by a single person in PDP-11 assembly language; assembly language is a very simple abstraction above machine code which does not facilitate higher level reasoning about its operations. The programmer left AECL in 1986 and was unable to be contacted since. There was nearly no documentation for the code, not in specifications, or for testing. Under deposition, the quality assurance manager for AECL stated that a "small amount" of testing was done on a simulator, with most effort done in system testing. AECL's safety analysis performed in 1983 excluded the software, assuming that:

1. Extensive simulation and field testing had effectively eliminated software errors.
2. The software does not degrade due to normal use or reproduction.
3. Computer errors are caused by hardware faults and radiation.

An independent safety analysis performed in 1987 foudn that software inspection was unable to provide a high level of confidence in the reliability of scanning and energy selection functions, due to the complexity of the code and time limitations of the inspection process. The conclusion of the 1987 safety analysis called for interlocks to back up the software.

The simple fact is that reproduction of the software bugs affecting the Therac-25 were extraordinarily hard to reproduce by simple testing methodologies. The bugs became apparent over time as the operators became increasingly proficient, allowing them to set up the machine faster. This speed of operation was a key factor in manifesting race conditions in the software. The Therac-25 so frequently had malfunctions that operators became numb to the problem. One operator testified, "It was not out of the ordinary for something to stop the machine... I cannot remember all the reasons it would stop, but there [were] a lot of them." The operators had been taught that it was impossible for the machine to overdose a patient, which AECL repeated during inquiries made in the cases of Katie Yarbrough, Frances Hill, Janis Tilman, and Voyne Ray Cox.

The case of the Therac-25 is illustrative of the complexity of systems. Although the immediate cause of the accidents was due to software failures, more accidents continued to happen for lack of of follow-through on reported incidents, overconfidence in the software, insufficient testing and analysis, and the manufacturer's bad faith. In particular, these accidents highlight the need for robust system designs with redundant fail safe and interlock mechanisms. Skepticism should be applied in the face of safety analyses -- for example, the claim of five orders of magnitude greater safety. Nancy Leveson and Clark S. Turner conclude their report by noting that software engineering principles had been violated -- documentation, software quality assurance practices and standards, simple designs, error reporting and software audit trails, and extensive testing and formal analysis at the module and software level.

Echoing the findings of the Ariane V report by Lions, Leveson and Turner state about code reuse:

Furthermore, important lessons about software reuse can be found here. A naive assumption is often made that reusing software or using commercial off-the-shelf software increases safety because the software has been exercised extensively. Reusing software modules does not guarantee safety in the new system to which they are transferred and sometimes leads to awkward and dangerous designs.

The Problems of AI in Software Engineering

An Increased Mental Load on Software Engineers

Unfortunately, Garry Tan is not lying about the incredible output of LLM models. There are massive costs to bear for the humans who must review such output, if they want to know that what has been produced is correct. Of course, the most fervent advocates for vibe coding insist that you shouldn't review the code yourself, and to automate review.

In a study performed by Alami et al. from the University of Southern Denmark, researchers found that "psychological costs arise among practitioners irrespective of their level of AI adoption ." In their review of related work, they relate:

• "users may perceive value even when measured task outcomes do not show corresponding improvement... perceived productivity is consequential, it shapes continued adoption, but it cannot be treated as equivalent to task performance, code quality, or organizational value ."
• "Fan et al. explicitly examine verification load and fatigue:" Fatigue and stress rose more for the AI users than the control group. "AI shows a steeper fatigue slope (KSS Δ = +0.70 vs Control +0.48) ."
• "...engineering effort is redistributed rather than displaced ."
• "Techno-overload appears when AI increases expected expected throughput or creates more output to inspect."
• "Emerging SE research questions the field's dominant emphasis on whether GenAI improves developer productivity."
• "Although GenAI tools are often framed as personal productivity aids, their widespread and largely individual use introduces new risks to the social interaction and collaboration of software teams."

Alami et al. found that participants in their study experienced AI as a disruptive force, with psychological costs emerging from adopting AI tools in their own workflows and organizational AI adoption. One of the key problems found in the study was that rapid evolution in the AI tooling frequently invalidated the skillsets users developed to interact with AI, developing into "uncertainty distress." In their conclusion, they state that there the "Verification tax reveals that while AI reduces the effort required to generate software artifacts, it simultaneously increases the cognitive effort required to understand, justify, and remain accountable for them."

Software engineers face increased work loads with a cost borne upon them as an increased cognitive load. This stress can impact their health, but it seems unlikely that they will receive additional compensation for their higher work loads and outputs. Worse, on top of the increased cognitive load, AI use negatively impacts learning.

AI Harms Learning and Causes Skill Atrophy

It is very troublesome that so many college students are using AI to cheat. A study conducted in the spring of 2024 by UC Berkeley found that 9% of students who used AI reported using it to cheat, while The Harvard Crimson reports that 25% of students used AI to cheat, and 33% of students used AI against instructor permission. Researchers from the University of California and McGraw Hill observed that while AI use can assist problem solving in the short term, students who used it were less capable of retaining knowledge when being assessed on math problems. This is one of the problems I don't see any ready solution for. We will have to learn new methods of finding and training desirable candidates to overcome the deficiencies of their education. Once they are hired, however, they will have to worry about building new skills and retaining them.

Engineers in the work force know that learning is a continuous process as part of your career. In a study performed by Anthropic, researchers Shen and Tamkin found that the level of skill formation gained by completing a programming task was significantly reduced . They state that their main finding is that "using AI to complete tasks that require a new skill (i.e. knowledge of a new Python library) reduces skill formation." Michels et al cite three empirical studies about skill atrophy :

• Microsoft Research found that 'Higher confidence in AI tools was associated with reduced critical thinking, while higher self-confidence in one's own skills was associated with maintained critical thinking.
• The Anthropic study I just mentioned, noting also that the speedup found was not significant. Michels et al. further state 'Anthropic's own framing acknowledges the trade-off: "productivity benefits may come at the cost of skills necessary to validate AI written code if the junior engineers' skill development has been stunted by using AI in the first place."
• A Frontier in Psychology study of university students found that AI dependence "significantly increased cognitive inertia, which in turn significantly reduced innovation capability ." Cognitive inertia refers to individuals' systematic avoidance of an in-depth processing when facing complex or open-ended tasks, tending to rely on heuristics or ready-made answers.

Together, these findings show that AI use is associated with less critical thinking and reduced skill formation.

A pre-review study performed by researchers from MIT performed a study in which participants wrote an essay. One group of participants used an LLM to write the essay, a second group used a search engine, and the third group did not use any tools to write the essay. The researchers found that "the most consistent and significant behavioral divergence between the groups was observed in the ability to quote one's own essay. LLM users significantly underperformed in this domain, with 83% of participants (15/18) reporting difficulty quoting in Session 1, and none providing correct quotes ." The researchers conclude that:

The LLM undeniably reduced the friction involved in answering participants' questions compared to the Search Engine. However, this convenience came at a cognitive cost, diminishing users' inclination to critically evaluate the LLM's output or 'opinions' (probabilistic answers based on the training datasets). This highlights a concerning evolution of the 'echo chamber' effect: rather than disappearing, it has adapted to shape user exposure through algorithmically curated content. What is ranked as 'top' is ultimately influenced by the priorities of the LLM's shareholders.

Communication Breakdown

One of the most detestable conditions arisen from embracing vibe coding has been the reduction of communication in favor of querying AI. Project managers are telling their engineers not to bother each other with questions, and to ask their AI instead.

Fred Brooks uses the Tower of Babel in the Mythical Man Month as an allegorical example of a project brought to failure by lack of communication:

...why did the project fail? Where did they lack? In two respects—communication, and its consequent, organization. They were unable to talk with each other; hence they could not coordinate. When coordination failed, work ground to a halt. Reading between the lines we gather that lack of communication led to disputes, bad feelings, and group jealousies. Shortly the clans began to move apart, preferring isolation to wrangling. -- Fred Brooks

More famously, he relates this formulation of communication channels as a graph: "If there are n workers on a project, there are (n²-n)/2 interfaces across which there may be communication, and there are potentially almost 2ⁿ teams within which coordination must occur." What does this equation imply when each person is using at least one AI agent to produce their work? What does it mean when the AIs have limited context windows, and that models and agents are replaced over time? There is an exponential explosion in coordination required. Moreover, the result of many AIs producing vibe code is a mosaic, reflective of Conway's Law:

The basic thesis of this article is that organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. We have seen that this fact has important implications for the management of system design. Primarily, we have found a criterion for the structuring of design organizations: a design effort should be organized according to the need for communication. -- Melvin E. Conway

The cost of a lack of communication is that each individual becomes more isolated, and fewer people know how to collaborate to build an understanding of the entire program under development.

Impacts on Free and Open Source Software

In the realm of open source software¹¹, Daniel Stenberg has reported on the pressure of AI generated bug reports. Sadly, it seems this is a new cost of business for open source code. Many open source projects are bombarded at higher rates than ever before with a sea of bug reports and issues, and no ability to rapidly filter high quality reports from low quality ones, and the severity of the issues. GitHub itself has greatly deteriorated as a service, if not because of being written AI itself, then due to strain on its resources by the tremendous volume of vibe coded commits of large volumes of code, and large testing campaigns. Moving off of GitHub may solve that problem, but leaves questions of where to host the code. There are alternative software forges¹² such as GitLab (which has also embraced AI), CodeBerg, and sourcehut, as well as options to self-host with solutions as minimal as cgit or Forgejo. Self-hosting imposes the burden of dealing with rampant scrapers which do not respect robots.txt¹³: this has vexed projects as important as the linux kernel itself: "we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones". Open source developers are strained under the pressure, and backing away from the disclosure practices for CVEs normalized over the last couple of decades. Open source software is often a critical component of proprietary commercial software, either directly, or in the technology stack used to produce and maintain it.

Deceptive Terminology, Reckless Hype, and Misleading Metaphors

The AI companies are very fond of giving concepts anthropomorphic names: agent, agentic, agentic engineering, Anthropic, attention, Claude¹⁴, copilot, hallucination, inference, learning, planning, reasoning, skills, soul, thinking, training, understanding, and probably many more. AI advocates frequently compare frontier AI models to holders of advanced degrees like PhDs. There are numerous stories of employees quitting their job or "whistleblowing" to the press, stating that the AI has gone rogue, or achieved human intelligence, or that the companies are on the brink of accomplishing superintelligence. Ted Chiang¹⁵ highlights the hype:

Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”

This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic’s CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution) said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.”

In particular, there are rampant claims of achieving human-like intelligence, or being on the verge of achieving so-called AGI (artificial general intelligence). These claims of consciousness and real intelligence have convinced many people, as has demonstration of feats that are extraordinarily difficult for humans to accomplish. However, I will explain soon why this is false. The matter is not helped by gasping claims of AI researchers claiming that models are already intelligent or soon-to-be:

• In June 2022, Google engineer Blake Lemoine claimed that Google's LaMDA model had gained consciousness. Google dismissed Lemoine, and stated that there was no evidence that LaMDA was sentient.
• In May 2023, Google employee Geoffrey Hinton quit Google, citing fears that the technology could become smarter than people, along with more well-founded fears of LLMs generating and running code on their own, use in autonomous weapons, and upending the job market.
• In November 2024, Rosie Campbell quit OpenAI, sharing a screenshot of her last message to the company in which she said "Take seriously the prospect that our current approach to safety might not be sufficient for the vastly more powerful systems we think could arrive this decade."
• In January 2025, OpenAI employee Steven Adler announced he quit in mid-November, claiming that the AGI race is a "very risky gamble," and that he was "pretty terrified by the pace of AI development."
• In September 2026, Jacob Coxon of Anthropic resigned, stating neither OpenAI or Anthropic are "acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives ."

These events represent a pattern of behavior, that seem closer to marketing stunts than real warnings. Citations of both Sam Altman of OpenAI and Dario Amodei of Anthropic claiming impending AGI are so abundant that I do not think it is necessary to even cite them. There is certainly a benefit for the players involved: increased fear of AGI signals to investors that it is more likely to be real, and encourages them to invest. Melanie Mitchell writes that using metaphors like "going rogue" and "losing control" is inappropriate, and "can lead to ill-formed decisions about how to fix problems or set policy."

AI advocates have used a variety of metaphors: role-playing actors, like using a spell-checker, like the calculator replacing the slide rule, like an alien mind, or even simply "just a tool." Douglas Hofstadter¹⁶ finds that perceptions are dependent on the processes of analogies. Indeed, the power of analogies is well known in psychological research, with particular applications in marketing: Goode et al. propose that "the power of analogical thinking to communicate functional knowledge has been well established in prior literature (e.g. Gregan-Paxtan and Moreau 2003; Roehm and Sternthal 2001)," and propose that analogy is also a powerful marketing tool to influence attitudes about product experiences. A frequent refrain of vibe coding proponents is that vibe coding is to programming as the calculator is to mental or on-paper arithmetic. The purpose of this paper is to refute that sort of claim, but it is important to note here that the nature of this very phrase is to influence the attitude of its recipients positively towards AI; after all, you don't do arithmetic by hand, do you? That is the question which such an analogy begs. Goode et al. state that by "comparing a product to a familiar but disparate experience, an analogy has the power to focus consumers on the evaluative, emotional, and multisensory information associated with a product experience." Perhaps the most frequently abused analogy is to consider AI "just a tool."

Melanie Mitchell, herself a student of Hofstadter, does not buy Sam Altman's exhortations that AI is just a tool. She states that the AI companies have designed their AI models like Claude and ChatGPT to exploit human psychology to think of them as a mind. Iris Meredith further argues that the argument of "just a tool" is inherently faulty because it ignores how the tool reshapes its user. Moreover, tools allow the user to do things that you would otherwise not be possible for them. As someone uses a tool, it becomes an extension of themselves (in the sense of Heidegger's Gestell), and they start viewing the world in terms of how to accommodate the tool, which reshapes cultural norms and attitudes. An illustrative example can be seen in how cars -- which are just a tool -- have shaped American cities and lifestyles, and even form a part of the American cultural identity. Meredith argues that tools should be carefully chosen to minimize the damage we do, and to promote a more harmonious society. Frank Elavsky further elaborates that tools aren't neutral, since their design enforces them to be used in specific ways. Frank writes that "artificial intelligence is the quintessential tool-as-a-drug," marketed as if there are no down-sides to using it. He feels the greatest cost is that since using AI reduces our struggles, it removes a large aspect of our greater humanity. I argue that in engineering, this cost is manifested in our loss of understanding of the things that we create.

Prompt Engineering and Disownership of Work

TODO flesh this section out.

I do not care for the term prompt engineering, since I feel like it diminishes what It is rather pompous to compare the activities of prompting and directing agents to any form of engineering. Some people sneer that Software Engineering itself is not a proper discipline of Engineering. I should not need to point to the NATO conferences on Software Engineering held in 1968 and 1969 again, but those are where the term first gained acceptance.

As noted in Kosmyna et al.'s study, the AI participants were the most likely to assign either partial credit in ownership to the AI, or no credit at all in ownership to themselves. Anecdotally, I have observed vibe coders crediting their tools instead of themselves, to my dismay.

The Essence of Engineering

What does it mean to be an engineer? It's easy to start by considering what engineers do. Engineers apply scientific knowledge and mathematics to design, create, and maintain products (e.g., gears, ladders, propellers) and systems (e.g., HVAC systems, satellites, nuclear power plants). Engineers practice critical thinking

Prior to getting a bachelor's of science in Aerospace Engineering, I enlisted in the United States Navy as a nuclear operator, specifically a Machinist's Mate. The naval nuclear program was created by Admiral Hyman G. Rickover, whose domineering will brought the first nuclear powered vessel, the U.S.S. Nautilus, to life in only 3 years. He required the utmost quality from reactor operators in skill: "But having a vocation means more than punching a timeclock. One must guard against banality, ineptitude, incompetence, and mediocrity."

We as a people seem inclined to accept average or mediocre performance. Mediocrity can destroy us just as surely as perils far more famous. It is important that we remember to distinguish between what it means to fail at a task and what it means to be mediocre. There is all the difference in the world between the life lived with dignity and style which ends up failing, and one which achieves power and glory yet is dull, unoriginal, unreflective, and mediocre. In a real sense, what matters is not so much whether we make a lot of money, hold a prestigious job, or whether we don’t; what matters is that we become people who seek out others with knowledge and enthusiasm—that we become people who can enjoy our own company. In the end, learning to avoid mediocrity gives all of us the chance to discover that success really comes in making ourselves into educated individuals, able to recognize that there is always a difference between living with excellence and living with mediocrity....

It is a device of the devil to let sloth into the world. One cannot work at his profession alone without stagnating: some time away from the job can be as much or more important in man’s overall development. Intellectually, we must never stop growing... Our minds must be forever skeptical yet questioning. We must strive to be singularly free from that failing so common to man, deplored by Pascal in the "Pensées," of filling our leisure with meaningless distractions so as to preclude the necessity of thought. To be an intellectual in the fullest sense, one’s mind must be in constant movement.

Intelligence—the central virtue of moral life—is being able to judge the limitations of knowledge. Though there is no substitute for intelligence, it is not enough. People may be intelligent but lack the courage to act. To find a purpose to life, one must be willing to act, to put excellence in one’s work and concern for what is right before personal safety.

No professional man has the right to prefer his own personal peace to the happiness of mankind; his place and his duty are to be in the frontline of struggling men, not in the unperturbed ranks of those who keep themselves aloof from life. If a profession is to have its proper place in the further development of society, it must be increasingly dissatisfied with things as they are....

The deepest joy in life is to be creative. To find an undeveloped situation, to see the possibilities, to decide upon a course of action, and then devote the whole of one’s resources to carrying it out, even if it means battling against the stream of contemporary opinion, is a satisfaction in comparison with which superficial pleasures are trivial. But to create, you must care. You must be willing to speak out.

A certain measure of courage in the private citizen is necessary to the good conduct of the State. Otherwise men who have power through riches, intrigue, or office will administer the State at will and ultimately to their private advantage. For the citizen, this courage means a frank exposition of a problem and a decrying of the excesses of power. It takes courage to do this, for in our polite society frank speech is discouraged. But when this attitude relates to questions involving the welfare or survival of the Nation, it is singularly unfitting to remain evasive. It is not only possible, but in fact the duty of everyone to state precisely what his knowledge and conscience compel him to say. Only complete candor and frankness, deep respect for the facts, however unpleasant and uncomfortable, great efforts to know them where they are not readily available, and drawing conclusions guided only by rigorous logic can bring many of today’s problems forward. -- Hyman G. Rickover

Moreover, Rickover demanded the highest standards of integrity from his sailors, and lamented what he perceived as a decline in ethical behavior:

The sense of responsibility for doing a job right seems to be declining. In fact, the phrase "I am not responsible" has become a somewhat standard response in our society to complaints of a breakdown in the system. This response is a semantic error. Generally what a person means is: "I cannot be held legally liable." Yet, from a moral or ethical point of view the statement is quite true. The person or organization taking this way out is truly not responsible; he is irresponsible.

The unwillingness to act and to accept responsibility is a symptom of America’s growing self-satisfaction with the status quo. The result is a paralysis of the spirit, entirely uncharacteristic of Americans during the previous stages of their history. Even the complaints about high taxes and high prices are illusory. Behind them is hidden the reality that the majority, in terms of sheer creature comfort, never had it so good. Those who are still on the outside looking in are not strong or numerous enough to make a political difference.

A major reason why so large a majority is smugly docile is that it has accepted the unwritten rules of game: Don’t rock the boat as long as you get your cut. Why become worked up over corruption as long as there are enough benefits of the fallout to go around? Once the acceptance of corruption becomes sufficiently widespread, effective exposure seems threatening to too many people and interests. Clamor for closing loop­holes declines in direct proportion to the number of people who benefit from loopholes of their own. Freedom of speech seems less important when the majority persuades itself that it is not likely ever to want to speak out to complain....

We shall not get through our troubles safely until a considerable number of Americans acknowledge that they themselves are a part of the process by which the society will be made whole.

The crucial element of hope amid the current drift lies in the recollection of what the American spirit of activist optimism has so often accomplished—when it was rallied by its leaders and philosophers to a vision of shared goals.

For the person who strives to excel, to shoulder responsibility and to speak out, there is an enemy wherever he turns. The enemy is a man who has a total willingness to delegate his worries about the world to officialdom. He assumes that only the people in authority are in a position to know and act. He believes that if vital information essential to the making of public decisions is withheld, it can only be for a good reason. If a problem is wholly or partially scientific in nature, he will ask no questions even though the consequences of the problem are political or social.

The enemy is any man whose only concern about the world is that it stay in one piece during his own lifetime. He is invariably successful and regards his good fortune not as a challenge to get close to the real problems of the age, but as proof of the correctness of everything he does. Nothing to him is less important than the shape of things to come or the needs of the next generation. Talk of the legacy of the past or of human destiny leaves him cold. Historically, he is the disconnected man. He is an enemy because he is detached from the kind of concern for the rights of unborn legions that will enable the world itself to become connected and whole.

To struggle against these enemies, and against apathy and mediocrity, is to find the purpose in life. -- Hyman G. Rickover

Education

The architect should be equipped with knowledge of many branches of study and varied kinds of learning, for it is by his judgement that all work done by the other arts is put to test. This knowledge is the child of practice and theory. Practice is the continuous and regular exercise of employment where manual work is done with any necessary material according to the design of a drawing. Theory, on the other hand, is the ability to demonstrate and explain the productions of dexterity on the principles of proportion. -- Vitruvius

Education. There are five basic goals for education. When I say "basic,” I mean goals that apply to everyone—men or women, rich or poor.

Goal 1: To learn to understand, appreciate, and take care of the natural world we live in....

Goal 2: To understand, appreciate, and learn to live with the fellow inhabitants of our planet....

Goal 3: To have an area of esthetic experience...

Goal 4: To earn a living....

Goal 5: To learn some kind of critical or intellectual method. I regard this as the most important of all. -- Hyman G. Rickover

We can get good designs by following good practices instead of poor ones. Good design practices can be taught. Programmers are among the most intelligent part of the population, so they can learn good practice. Thus a major thrust in the United States is to promulgate good modern practice. New curricula, new literature, new organizations such as the Software Engineering Institute, all have come into being in order to raise the level of our practice from poor to good. This is entirely proper. -- Fred Brooks

Ethics

As for philosophy, it makes an architect high-minded and not self-assuming, but rather renders him courteous, just, and honest without avariciousness. This is very important, for no work can be rightly done without honesty and incorruptibility. Let him not be grasping nor have his mind preoccupied with the idea of receiving perquisites, but let him with dignity keep up his position by cherishing a good reputation. -- Vitruvius

Morals and Ethics. There is abundant evidence around us for one to conclude that morals and ethics are becoming less prevalent in people’s lives. The standards of conduct which lay deeply buried in accepted thought for centuries are no longer absolute. People seem unable to differentiate between physical relief and moral satisfaction. Many confuse material success in life with virtue. -- Hyman G. Rickover

I think the time will come when the United States is not as prosperous as it is today and when citizens will be forced to think through what is going on....

A preoccupation with the so-called bottom line of profit and loss statements, coupled with a lust for expansion, is creating an environment in which fewer businessmen honor traditional values; where responsibility is increasingly disassociated from the exercise of power; where skill in financial manipulation is valued more than actual knowledge and experience in the business; where attention and effort is directed mostly to short-term considerations, regardless of long range consequences.

Political and economic power is increasingly being concentrated among a few large corporations and their officers --

AI does not think, and is not Human

There is a long history in the development of AI. Perhaps ironically, many of the figures I have quoted in this paper, particularly those from the NATO conferences, have made significant contributions to the field. Arguably, the contemporary foundations of it can be found in Alan Turing's work. No discussion on this matter cannot start without an examination of the ubiquitously referenced Turing test.

The Turing Test

First described in Alan Turing's article "Computing Machinery and Intelligence," as the imitation game, the test is posed as follows:

I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous... Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.

The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A."

The point of the game is to determine if a machine can fool an interrogator when given sufficient masking to conceal its identity. Turing explicitly states that the question, 'Can machines think?', is "too meaningless to deserve discussion." He poses a series of potential objections, and his rebuttals to these objections, which we will revisit in an examination of the works of John Searle and Peter Naur. Turing gives an early description of machine learning which still has relevancy in modern AI technologies, and presents a particular theory of mind by manner of analogies. Ultimately, Turing turns his attention towards simulating a brain with a computer. He conceives of trying to simulate a child's mind which could then be taught, likening it to a blank notebook ready to be programmed. He describes an evolutionary model to develop the simulated brain of a child. It begs questions of ethics which he has completely shunned. He even gives Helen Keller as an example that "education can take place provided that communication in both directions between teacher and pupil can take place by some means or other."

It is quite evident that Turing had an extremely mechanical view of reality, and there is scarce evidence of sympathy, and no question of the ethics involved. What happens to people if artificial intelligence is developed? If truly intelligent machines were developed, what moral implications are there for applying evolutionary selection on them, and using punishment to teach them? The latter of these questions is merely hypothetical, while the first is one of the most pressing issues of our time.

John Searle's Chinese Room

The Chinese Room is perhaps the most famous critique of the Turing test. John Searle articulated the Chinese Room thought experiment in his article "Minds, Brains, and Programs" as such: imagine locking a person who only speaks English into a room, and interrogators outside the room who only speak Chinese. The person is given sets of Chinese texts, and instructions in English. The interrogators pass questions written in Chinese about the texts under the door, and by using the English instructions, the person can select from the other Chinese texts to create answers passed back under the door which make sense and are grammatical to the Chinese interrogators. Despite appearances, the person in the room does not actually have any understanding of Chinese; there is a fundamental difference in following instructions to produce these answers compared to the same scenario in English. By Searle's reckoning, operating on formally specified elements computationally does not provide a basis of understanding. Strikingly, Searle touches on another article which I discuss elsewhere in this paper, on the nature of tools: "in artifacts we extend our own intentionality; our tools are extensions of our purposes, and so we find it natural to make metaphorical attributions of intentionality to them." In other words, we frequently anthropomorphize things because we're projecting onto them.

Searle argues that the difference between natural understanding and formal rule following in the Chinese Room, is that by the former, there are meaningful associations presented in the texts, questions, and answers, while it is only logic present in the latter. Simulation of thought is not the same thing as thought, and the computers running a program to answer questions lack intentionality. Searle states that human interpretation is ultimately the source of meaning in computer programs: "such intentionality as computers appear to have is solely in the minds of those who program them and those who use them, those who send in the input and those who interpret the output." He regards the arguments of the AI proponents of his day as implicitly resting upon Cartesian dualism, where mind is separate from body:

Unless you believe that the mind is separable from the brain both conceptually and empirically -- dualism in a strong form -- you cannot hope to reproduce the mental by writing and running programs since programs must be independent of brains or any other particular forms of instantiation. -- John Searle

For Searle, the phenomena of the mind are the consequence of operation of the brain, but a program is not a product of the computer on which it executes: programs are not conscious.

Peter Naur's Arguments Against The Turing Test

Peter Naur, as has been previously noted from his appearance at the NATO conferences, was a computer scientist. Originally trained as an astronomer, Naur switched to the nascent field of computer science (or datalogy, as he preferred to call it), and developed his beliefs through the lenses of psychology and linguistics, being profoundly influenced by the American philosopher and psychologist William James, and the Danish linguist, Otto Jesperson. Unlike many of his contemporaries, Naur was outspoken on the negative impacts of computers on people. Naur gave a speech about the development of AI on Danmarks Radio in 1966, telling a story familiar to us now: "One conflict touching deeply philosophical questions is about what the programmed computers may be made to do." He recites the work of AI scientists to conquer chess, perform automatic language translation, prove mathematical theorems, and achieve computer vision, and the ultimate goal of achieving what we now call Artificial General Intelligence.

In conflict with these views the critics of artificial intelligence maintain that it is wrong to claim that a human being can be seen as a machine controlled by rules, and that those who conjure up the fantastic visions are the victims of deep delusions. -- Peter Naur

Unfortunately, it seems modern critics are much more quiet about the reduction of humans to machines.

Naur addresses the Turing test by considering the question originally posed, but left unanswered, by Turing: "Can machines think?" using linguistic analysis to dissect Turing's lack of consideration for what the words "think" and "can" mean. He decides that of 3 possible senses for 'can,' that Turing simply meant in the sense of 'being able to.' When it comes to the word 'think', Naur frames it in William James' notion of thought, "we must simply say that thought goes on." William James elaborates on this by characterizing thought as a continuously changing part of a personal consciousness, something which "always appears to deal with objects independent of itself," and which focuses on a subset of objects by filtering others out. Naur relates this back to the question of 'Can X think?,' and decides that it is in fact a meaningless question since thought is a process that goes on, rather than an action which is done.

Naur then goes on to describe thinking as an intellectual exercise. He cites Gilbert Ryle's "The Concept of Mind": as an attack on the Cartesian duality of body and mind. Ryle's work examines that the Cartesian notion of mind causes linguistic confusion. Naur summarizes Ryle's work:

1. Being intelligent is not the same as possessing knowledge.
2. Behavioral rules cannot describe intelligent behavior, and the actions of an intelligent person cannot be modeled as someone simply following rules, or else it would devolve to a recursive, circular dependency.
3. Intellectual activity is characterized by a person's development of theories, which prepares them to act.
4. Terms such as 'judgment,' 'inference,' and 'conception,' are representative of the linguistic results of intellectual activity, but are not representative the actions of intelligence itself.

From this overview, Naur concludes that the question 'Can X think?' is meaningless, "since what is characteristic of intellectual activity is neither a kind of operation nor a corresponding ability."

Naur then examines Turing's imitation game using this characterization of intelligence. He notes that, as formulated by Turing, the purpose of the game is for one of the participants to convince an evaluator that he (when played by a man), or it (when played by a computer), is a woman, while the other participant must convince the evaluator that they are in fact the woman. Naur believes that with good actors, this is hard enough to decide in a short period, but becomes more likely with sufficient time to identify inconsistencies in the feigned identity. He raises the suspicion that Turing was motivated to choose this particular form of the game instead of a simpler variant of the interrogator identifying if a lone participant is a person or a machine because he thought that would be too difficult for the machine, and would remain so "even a hundred years from now." I think time has proven Naur's prediction wrong, but he still has more insight to share. He suggests the inverse of the Turing game, where a machine acts as a judge instead. With a machine as an evaluator, the same sort of rules used to conceal player identities are still required, to prevent the machine from judging based on timing characteristics or computational speed and accuracy. Where the human evaluator is deprived clues like facial expression, the machine judge would have to have answers relayed through human intermediaries to induce an artificial time delay in the response. Naur postulates that by reducing the communication mechanisms to writing, it entices intellectuals into believing that the machine is truly intelligent by the cultural esteem given to writing. I think this is one of the most insightful and lasting contributions to the discussion that Naur has given us.

Naur also critically analyzes Turing's paper to determine what Turing's notion of thinking was. Naur believes that Turing assumes that thinking is "the same as exercise of intellectual capacity," that consciousness is separate from body, and that thinking is an action. Naur notes that Turing believes that if under objection (4)¹⁷, the game is rejected, that it "must imply acceptance of an extreme solipsist position," which reinforces that Turing believes in Cartesian dualism. Naur also critiques the viva voce reply to (4) that Turing gives, because part of such an examination is that the interviewer witnesses the interviewee's facial expressions, voice, and body language -- all of which are representative of the thought process of the interviewee and detectable by human interviewers instinctively -- and the stipulations of the imitation game prohibits the interviewer from observing these non-linguistic communication methods. Naur concludes that by understanding consciousness "as a basic life experience of mature human beings, beyond the reach of any finite test," the imitation game can be rejected. Summarizing:

1. Thinking is not a specific thing that a human being can or cannot do, and the phrase 'Can X think?' is meaningless for any X.
2. The restrictions on communication in the Turing test are arbitrary and have no relation to any human characteristics.
3. "The awareness that each of us has of our fellow beings' nature, including the similarity of that nature with that of our own, is part of our whole life experience, and cannot be reduced to, or expressed in terms of, any particular tests or games."

Naur has more parting words on the topic in his overview of his corpus, "Computing Vs Human Thinking", which takes the same opinion as Searle:

And so, as the conclusion of this discussion of the theme given in the title: Computing presents us a form of description. This form is very useful for describing a great variety of phenomena of this world, but human thinking is not one of them, the reason being that human thinking basically is a matter of the plasticity of the elements of the nervous system, while computers — Turing Machines — have no plastic elements. For describing human thinking one needs a very different, non-digital form, as demonstrated by the Synapse-State Theory. -- Peter Naur

The Eliza Effect

Ted Chiang argues that the way we interact with LLMs produces a powerful effect where the user gets the impression that they are having a real dialogue with a conscious entity, which would satisfy the Turing test: "it's simply the fact that LLMs emit grammatical sentences and we are accustomed to reading intention into sentences." This phenomenon has existed since the mid-1960s, when it was called the Eliza Effect. Douglas Hofstadter defines the Eliza Effect as "the susceptibility of people to read far more understanding than is warranted into strings of symbols -- especially words -- strung together by computer." The Eliza Effect is a perennial problem for computer users ever since the original ELIZA system from which it took its name, affecting journalists, computer scientists, mathematicians, and philosophers alike. Hofstadter reiterates, as Searle and Naur, that the misunderstanding arises because the human users confused the output of a program operating on strings for understanding the real world domain on which those strings represent meanings relevant to the users; this misunderstanding often benefits AI researchers.¹⁸

Hofstadter brings up a particular example of an AI program whose authors (Langley et al.) claimed "discovers empirical laws," called BACON. Hofstadter declares that the discoveries "made" by BACON were insubstantial because it was given only the data necessary to derive these discoveries represented in near-optimal form, which the Langley et al. used in conjunction with heuristics to "derive" Kepler's 3rd law. Like Hofstadter, Albrecht Heeffer was similarly unimpressed: 'The authors admit that the input data "were contrived to fit Kepler's law exactly"... one cannot escape that the data has been contrived and procedures have been arranged ad hoc to discover what is already known..' Heeffer was especially dubious since the data used to "discover" Kepler's third law can be used in infinitely many quantitative ways, making the combination of the precise formulation of Kepler's third law very suspicious. Hofstadter drives home that Kepler lived in a pre-scientific world, and was originally working with a hypothesis that the motion of the planets related to Platonic solids and not, as he would ultimately determine, conic sections. Michael Gormon also notes in his review of Langley et al.'s work that "the program falls ludicrously short. No one presented Kepler with a table of data, neatly arranged, and told him to discover an arbitrary mathematical relationship." Hofstadter concludes his review of BACON with an observation that it is typical of AI research, which understates the importance of the work in representation-building.

Letters To A German Princess

In his letters to a German princess, Euler wrote a letter on "The True Foundation of human Knowledge." I have taken the liberty of omitting the introductory paragraph and the first sentence of the second paragraph, but the remainder of the letter is presented below, with some modifications to spelling and grammar which were not settled at the time of its translation to English:

We are very far from being always certain of the truth of all our sentiments; for we are but too frequently dazzled by appearances, sometimes exceedingly slight, and whose falsehood we afterwards discover. As we are, therefore, continually in danger of deceiving ourselves, a reasonable man is bound to use every effort to avoid error, though he may not always be so happy as to succeed.

The thing to be here chiefly considered is the solidity of the proofs on which we found our persuasion of any truth whatever, and it is absolutely necessary that we should be in a condition to judge if they are sufficient to convince us or not. For this effect I remark that, first, that all truths within our reach are referable to three classes, essentially distinguished from each other.

The first contains the truths of the senses; the second, those of the understanding; and the third, those of belief. Each of these classes requires peculiar proofs of the truths included in it, and in these three classes all human knowledge is comprehended.

Proofs are the first class are reducible to the senses, and are thus expressed:

This is true, for I saw it, or am convinced of it by the evidence of my senses.

It is thus I know that the magnet attracts iron, because I see it, and experience furnishes me with incontestable proofs of the fact. Truths of this class are called sensible, because they are founded on the senses, or on experience.

Proofs of the second class are founded in ratiocination; thus:

This is true, for I am able to demonstrate it on principles of just reasoning, or by fair syllogisms.

To this class, principally, logic is to be referred, which prescribes rules for reasoning consequentially. It is thus, we know, that the three angles of a rectilinear triangle are together equal to two right angles. IN this case I do not say I see it, or that my senses convince me of it; but I am assured of its truth by a process of reasoning. Truths of this class are called intellectual, and here we must rank all the truths of geometry, and of the other sciences, in as much as they are supported by demonstration. You must be sensible, that such truths are wholly different from those of the first class,in support of which we adduce no other proofs but the senses, or experience which assure us that the fact is so, though we may not know the cause of it. In the example of the magnet, we do not know how the attraction of iron is a necessary effect of the nature of the magnet, and of iron; but we are not the less convinced of the truth of the fact. Truths of first class are as certain as those of the second, though the proofs which we have of them are entirely different.

I proceed to the third class of truths, that of faith, which we believe, because persons worthy of credit relate them; or when we say:

This is true, for several creditable persons have assured us of it.

This class, accordingly, includes all historical truths. You believe, no doubt, that there was formerly a king of Macedon, called Alexander the Great, who made himself master of the kingdom of Persia, though you never saw him, and are unable to demonstrate, geometrically, that such a person ever existed. But we believe it on the authority of the authors, who have written his history, and we entertain no doubt of their fidelity. But may it not be possible that these authors have concerted to deceive us? We have every reason to reject such an insinuation, and we are as much convinced of the truth of these facts, at least of a great part of them, as of truths of the first and second classes.

The proofs of these classes of truths are extremely different; but if they are solid, each in its kind, they must equally produce conviction. You cannot possibly doubt that Russians and Austrians have been at Berlin, though you did not see them: this, then, is to you a truth of the third class, as you believe it on the report of others; but to me it is one of the first class, because I saw them, and conversed with the, and as many others were assured of their presence by means of other senses. You have, nevertheless, as complete conviction of the fact as we have.

-- Leonhard Euler

Conclusion

This paper was written without any assistance from any AI tools.

Further Reading

• Software Engineering: Report of a conference sponsored by the NATO Science Committee, 1968.
• Software Engineering Techniques: Report of a conference sponsored by the NATO Science Committee, 1969.
• Thoughts on Man's Purpose in Life...and Other Matters, Admiral Hyman G. Rickover, 1974. U.S. Naval Institute.
• The Mythical Man Month (2nd Edition), Fred Brooks, 1995.
• Programming as Theory Building, Peter Naur, 1985.
• Computing versus human thinking, Peter Naur, 2007.
• Computing: A human Activity, Peter Naur, 1992.
• Epigrams in Programming https://engineering.yale.edu/academic-study/departments/computer-science/about/epigrams-programming

Bibliography

Footnotes: