好工程不信任工程师：AI时代，代码不再是护城河

TLDR:

AI may not be the end of software engineering.

It may be the first thing forcing us to finally practise it.

I was talking with people who work in factories. Not people who write software for factories. People who spend every day next to machines, materials, operators, maintenance crews, and all the strange things that happen after a clean design meets the physical world.

They told me they do not trust software engineers.

They were not making an abstract point about education or job titles. They had seen engineers arrive with a model that worked perfectly on paper and failed to account for what actually happens on the floor. Dust gathers on a sensor. A replacement part comes from a slightly different batch. A guard is bypassed because it slows production. Heat, vibration, wear, and tired people change the conditions. A machine behaves differently on a cold morning than it did during acceptance testing.

The drawing can be correct. The code can be correct. The system can still be dangerous.

My first instinct was to defend engineers. I am called one, after all.

Then I realized that good engineering agrees with the factory workers.

NASA does not solve difficult problems by finding brilliant engineers and trusting them. Aviation does not accept software because its author is experienced. Nuclear plants do not rely on the developer to remember every way a system might fail.

They build processes around the assumption that engineers can be wrong.

Requirements can be wrong. Designs can be wrong. Tests can prove the wrong thing. Entire teams can share the same blind spot. This is why regulated engineering has hazard analysis, requirements traceability, operational validation, independent verification, safety reviews, and evidence that survives after the original engineer has left. NASA even defines independence for critical software across technical, managerial, and financial dimensions. (SWEHB)

Meanwhile, software engineers are having an identity crisis.

AI can write code. It can debug it, explain it, refactor it, generate tests, and review the resulting pull request. It may already be better than many of us at some bounded coding tasks, and it will not stop improving.

So software engineers are asking a frightening question:

If AI writes the code, what exactly is my job?

The question reveals more than we want it to.

If removing the keyboard also removes our engineering identity, perhaps coding was the identity all along.

The 1968 NATO conference that made the term software engineering famous was not organised to give programmers a more impressive title. It was a response to a crisis. Software systems were becoming larger, more expensive, less predictable, and harder to maintain. The term was an attempt to push software toward the discipline expected from engineering. (Newcastle Uni Comp Sci Homepages)

More than half a century later, we kept the word engineer, but much of the industry reduced the work to this:

ticket
↓
code
↓
tests
↓
pull request
↓
green CI
↓
production

The ticket contains a few sentences. The software engineer fills in the missing requirements while writing the implementation. The same engineer writes the tests and decides which cases matter. Someone from the same team reads the diff, usually with the same product context, the same architecture, the same deadline, and many of the same assumptions.

Then the pipeline turns green.

One small process has defined the truth, implemented the truth, and proved the truth.

ticket
↓
code
↓
tests
↓
pull request
↓
green CI
↓
production

We use words borrowed from engineering: architecture, design, reliability, infrastructure, incident, post-mortem. But often it is a shadow of engineering. The requirement is a Jira ticket. The safety argument is a pull-request comment. The design rationale lives in someone’s head. Validation means watching production metrics after release.

The code is treated as the only reality that matters.

This worked better when writing code was expensive. Implementation moved slowly enough that experienced people could hold a surprising amount of the system in their heads. A good developer could compensate for a weak process with memory, care, and judgment.

AI removes that protection. It can produce more changes than a human can understand line by line. It can turn one vague sentence into thousands of lines of plausible code before anyone has asked whether the sentence was true.

AI did not create software engineering’s identity crisis.

It exposed it.

NASA’s systems-engineering process does not begin with implementation. It begins with stakeholder expectations: who needs the system, how they intend to use it, where it will operate, what constraints exist, and what success means.

Only much later is a component realised by buying it, building it, reusing it, or coding it. In NASA’s own description, coding is one implementation method inside a larger engineering process. It is not the process itself. (NASA)

This is also why engineering separates verification from validation.

Verification asks whether we built the product according to its requirements.

Validation asks whether we built the right product for its intended purpose and environment.

Those questions sound almost identical until a perfectly verified system fails in operation. NASA explicitly distinguishes compliance with written requirements from proving that a product accomplishes its intended purpose in its intended environment. (NASA)

You can implement every requirement correctly and still build the wrong system.

You can also write the wrong requirement, implement it perfectly, achieve complete test coverage, and congratulate yourself when every test passes.

This is where the factory worker matters. Their experience is not an annoying opinion to collect after the design is finished. It is part of the engineering input.

When an operator says, “That valve sometimes sticks after the machine has been cold all night,” the engineering response should not be, “The specification says it does not.”

The response should be:

Our model is missing something.

The model does not get the final vote. The physical world does.

Most software requirements begin with desired behaviour:

The user should be able to open the valve.

Safety engineering begins with a different question:

What happens if the valve opens at the wrong time?

Then it keeps going.

What if it never opens? What if it opens twice? What if the sensor reading is stale? What if the command arrives three seconds late? What if the valve reports that it is closed when it is still open? What if the software works exactly as designed, but the operator misunderstands the display?

That creates a different flow:

operational reality
↓
possible hazards
↓
safety constraints
↓
requirements
↓
system design
↓
implementation
↓
verification
↓
evidence

NASA’s software-safety guidance says that preliminary hazard analysis identifies hazard causes and possible controls, which then become inputs to safety requirements. Its requirements go further: system hazard analyses and software safety analyses must create or identify the software requirements needed when software may cause, contribute to, mitigate, or control a hazard. (SWEHB)

operational reality
↓
possible hazards
↓
safety constraints
↓
requirements
↓
system design
↓
implementation
↓
verification
↓
evidence

This starts before implementation, but it is not a one-time document exercise. New information from design, testing, operations, and failures changes the hazard model. The requirements and evidence have to change with it.

Now consider a function like this:

func OpenValve() error

Nothing inside that signature tells you how much evidence it deserves.

You need to know what the valve controls. You need to understand the pressure, material, temperature, timing, failure modes, operator response, maintenance history, and whether an independent mechanism can stop the flow.

Failure might mean a delayed batch.

It might mean a destroyed machine.

It might mean a dead person.

The criticality of code lives outside the code.

This is another place where ordinary software often reverses engineering. We look at the diff and decide how risky the change feels. We count files, lines, dependencies, and services touched.

Engineering starts with the consequence.

In an engineering system, the requirement is the obligation.

The design is an argument for satisfying that obligation. The code is one realisation of the design. Tests, analysis, simulations, reviews, and operational observations are different forms of evidence.

The implementation can change while the obligation remains.

You can rewrite the system in another language. You can replace one algorithm with another. You can move from custom hardware to an off-the-shelf component. You can throw away the current test suite and build a better one.

The requirement should survive all of that.

Code and tests matter, but they are replaceable artifacts. They are not the reason the system exists, and they are not the final authority on what the system should do.

Mainstream software inverted this relationship.

The code became the source of truth. Tests became an explanation of the current code. Requirements became temporary prose that started rotting as soon as the ticket was closed.

Six months later, nobody knows whether a strange condition is intentional, defensive, obsolete, or accidental. We read the implementation and try to reconstruct the decision that produced it.

That is not traceability.

It is archaeology.

This matters even more with AI. Generated code can be internally consistent and still be based on the wrong intent. Generated tests can confirm the same misunderstanding. A generated explanation can make the whole mistake sound reasonable.

No model can prove a system against an intent that was never made explicit.

A normal software team often asks the developer to do all of these things:

interpret the requirement;

decide the design;

write the implementation;

select the tests;

write those tests;

explain why the change is safe.

A colleague then checks whether it all looks plausible.

This can be good work. It is not independent evidence.

NASA uses Independent Verification and Validation for critical software specifically to introduce a different perspective. Technical independence means the people doing the analysis were not involved in developing the system. Managerial independence lets them choose what to analyse and how. Financial independence protects the work from pressure by the development organisation. NASA’s rationale is direct: a genuinely different perspective can find subtle errors that the development team overlooks. (SWEHB)

Not every billing page needs an independent verification organisation. Rigor should follow risk.

But the principle matters:

The assumptions that created the system should not be the only assumptions used to prove it.

This does not change merely because agents are involved.

An agent can write the code, generate the tests, review the diff, and produce a confident safety summary. A second agent can review it. A third can vote on the result.

But if all three receive the same incomplete requirement, share the same context, and optimise for the same target, their agreement may not mean much.

Three agents agreeing can be one assumption repeated three times.

Even strong coverage does not fix an upstream mistake. NASA requires 100 percent MC/DC coverage for identified safety-critical software components, meaning each condition in a decision must be shown to affect the outcome independently. That is serious evidence about the implementation. It still cannot tell us that the requirement was correct or that the system is safe in its real environment. (SWEHB)

A green pipeline tells us that the checks we selected passed.

It does not tell us that we selected the right checks.

This is where the software-engineering identity crisis becomes useful.

If your idea of engineering is turning tickets into code, AI is coming directly for the centre of your identity. Coding faster will not solve that. Learning one more framework will not solve it. Becoming better at prompting a model may extend the same identity for a while, but it does not answer the question.

The answer is not to prove that humans will always write smarter code.

Some code is genuinely difficult. Some parts require deep performance work, hardware knowledge, novel algorithms, or careful human judgment. But AI is getting good at many of those tasks too. Building our professional identity around the remaining areas where humans currently outperform it is a shrinking defence.

The way out is to stop treating coding as the definition of engineering.

Engineering is understanding the system before choosing the implementation. It is making intent explicit. It is finding the constraints hidden in the environment. It is asking what can go wrong before somebody discovers the answer in production.

It is deciding what must always be true, what must never happen, how severe failure would be, which uncertainty remains acceptable, and what evidence is strong enough for the risk involved.

It is also deciding where an agent can act alone and where human judgment is required.

AI can help with all of this. It can propose hazards, formalise requirements, analyse designs, generate test cases, search for counterexamples, and inspect evidence. This is not an argument that humans own reasoning and machines should only type code.

The point is that engineering is not a task owned by one kind of worker. It is a system for turning uncertain intent into explicit obligations, and explicit obligations into evidence.

The code is part of that system.

It is not the system.

If the only thing separating us from a code generator was that we personally typed the code, we were not defending engineering. We were defending a temporary monopoly on construction.

The answer is not to make every software company imitate NASA.

Most software is not flight control. A change to button text does not need a hazard review, formal verification, and an independent assurance organisation. Copying every ceremony from a regulated industry would make ordinary development slower without making it meaningfully safer.

The important principle is proportionality.

A visual change may need a preview and a reviewer. An authentication change deserves stronger evidence. A destructive database operation should prove its safety conditions. Software controlling a medical device or physical machine belongs in another category entirely.

But every change should have some clear relationship between intent, implementation, and evidence.

That is the direction behind ReqProof.

A requirement should exist above the current code. It should say what must be true, under which conditions, and why. It should connect to hazards, constraints, implementation, tests, analysis, and operational evidence.

When something changes, we should be able to answer simple questions:

What obligation changed? Why did it change? Was the change made by a human or an agent? What evidence was produced? Which assumptions were challenged? What remains uncertain? Where was human approval required?

NASA’s FRET project demonstrates one part of this model. It lets engineers express requirements in structured natural language, gives those requirements precise semantics, and translates them into temporal logic for analysis. The important idea is not the syntax. It is that the requirement becomes something we can reason about, not prose that disappears after implementation. (NASA Technical Reports Server)

ReqProof takes that idea into an agent-driven software lifecycle.

The code may be generated. The tests may be generated. Parts of the analysis may be generated. But the obligation remains visible, and the evidence stays attached to the change.

We should not need to trust that an agent understood the requirement.

We should not need to trust that the developer remembered every constraint.

We should be able to inspect the argument.

So, are software engineers real engineers?

Some are. Some are programmers with a more expensive title. The same person may be doing engineering on one project and simply implementing features on another.

The language does not decide it. The material does not decide it. The complexity of the code does not decide it.

A better test is whether the organisation can explain what must be true, what can go wrong, why the design should work, and what evidence supports that claim—without asking us to trust the person who built it.

The factory workers were right. They should not have to trust an engineer’s clean model over what they see every day. Their knowledge should shape the requirements. Their experience should change the hazard analysis. Their objections should remain visible until someone produces evidence that they have been addressed.

Good engineering does not ask the factory floor to trust the engineer.

It gives the factory floor a way to prove the engineer wrong.

AI may not be the end of software engineering.

It may be the first thing forcing us to finally practise it.