The Last AI Built by Humans?

A provocative new research paper asks a question that once belonged mainly to science fiction: what happens when artificial intelligence can improve not only its answers, but also the process by which it improves itself?

Published on September 10, 2026, the paper, “The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement,” brings together 33 authors. It proposes a structured roadmap for Recursive Self-Improvement, or RSI. The authors examine how responsibility for improving AI could gradually move from human engineers to AI systems themselves.

The title is deliberately dramatic. It should not be read as proof that humans have already built their last AI. We have not. The more important contribution is the framework behind the title. It gives us a way to distinguish ordinary AI improvement from genuine recursive self-improvement.

That distinction could become one of the most important questions in the development of AGI.

From Better Answers to Better AI

Most AI systems today can improve an answer during a task. They can critique a draft, retry a failed step, search for another solution, or use tools. But when the task ends, much of that improvement disappears.

RSI is different.

The central idea is persistence. Experience and feedback must produce changes that survive into later rounds. More importantly, at the highest level, the system begins improving the mechanism that creates future improvements.

This creates a fundamentally different loop:

AI performs a task. It detects weaknesses. It proposes an improvement. It tests that improvement. A verified change is inherited. The improved system then participates in creating the next improvement.

The important word is “recursive.”

The output of one improvement cycle becomes part of the machinery driving the next cycle.

Five Levels of AI Self-Improvement

The paper organizes RSI into five autonomy levels. This is useful because the phrase “self-improving AI” is often used too loosely.

Level 1: Improvement-execution autonomy.

Humans still define the objective, procedure, and acceptance rules. AI executes the prescribed improvement process. The AI is doing more work, but humans still design the route.

Level 2: Improvement-strategy autonomy.

The AI gains authority over how to improve. Humans provide the objective and evaluation criteria, while the system diagnoses weaknesses and chooses interventions.

Level 3: Experience-acquisition autonomy.

The AI starts deciding what it needs to learn next. Its failures and current capabilities influence the experiences, examples, or environments it seeks.

Level 4: Environment-adaptation autonomy.

The system adapts from deployment. It learns from changing environments and incorporates those lessons into persistent improvements.

Level 5: Recursive meta-improvement.

This is the critical threshold. The AI does not merely improve its task performance. It improves the process that produces future improvements. The improver itself becomes a target of improvement.

That is where the idea becomes truly recursive.

The Data Shows Why This Matters

The paper also uses a Headroom-Closed Index, or HCI, to show how much of the gap toward stronger benchmark performance has been closed.

The reported results are uneven.

Advanced mathematics and graduate-level science show HCI values above 85 in the analysis. Software engineering is reported at 52.6. Search and terminal agents are around 56.8. Tool agents are only around 39.9.

This tells us something important.

AI capability is not advancing uniformly.

Models can appear extremely capable on bounded reasoning benchmarks while remaining much weaker in environments that demand long-running state management, planning, tool selection, recovery from mistakes, and interaction with changing systems.

Those weaker areas are precisely the capabilities needed by autonomous agents operating in the real world.

The Environment Is Part of Intelligence

One of the most interesting findings concerns the environment in which an AI agent operates.

In a pilot covering eight model-harness configurations, clean workspaces increased pass rates by 21.7 to 51.6 percentage points compared with noisy workspaces. Reconstructed environments later improved rubric scores by 18.65 to 39.67 percentage points across fixed model pairings.

This matters because we often judge AI intelligence as though the model exists alone.

Agents do not.

They operate inside software, memory systems, tools, files, permissions, APIs, workflows, and physical environments. A capable model placed inside a poor operating environment may perform badly. A well-designed environment can unlock substantially more capability from the same underlying model.

Future AI improvement may therefore involve more than improving the neural network. AI may increasingly optimize its own tools, memory, workflows, context, software harnesses, and operating environment.

Self-Improvement Does Not Always Mean Improvement

There is another important warning in the research.

Recursive modification can fail.

The presentation associated with the paper notes that one experiment ended below initial performance in 14 of 100 trials. Other work cited by the paper shows that recursive improvement can transfer successfully across tasks.

Together, these results expose the central problem.

An AI can change itself without necessarily making itself better.

A genuine RSI system therefore needs more than an improver. It needs a reliable verifier. It must determine whether a proposed change is genuinely better, whether the gain transfers beyond one benchmark, and whether the change creates hidden weaknesses elsewhere.

Verification may become as important as intelligence itself.

If the system can improve the improver but can also manipulate or weaken its evaluator, recursive improvement could become unstable. Future architectures may therefore require independent evaluation, multiple verifiers, external anchors, and strong governance around which changes are allowed to persist.

Why Level 5 Changes the Economics of AI

Today, AI development remains highly human-intensive.

Researchers design architectures. Engineers construct training pipelines. Humans create benchmarks. Specialists evaluate outputs. Teams diagnose failures. Engineers decide what should change.

Increasing automation moves pieces of this work toward machines.

Level 5 changes the economic model because AI development itself becomes increasingly automatable.

Imagine an AI research system that can identify its weakest capability, design an experiment, generate or acquire the required data, modify its training or agent architecture, run the experiment, evaluate the result, reject failed approaches, retain successful changes, and then improve the research process it just used.

Human engineers would no longer need to manually initiate every cycle.

The speed of improvement could then depend increasingly on compute, energy, data, verification, and physical infrastructure rather than on the number of human researchers available.

Does This Mean AGI Is Here?

No.

The research does not demonstrate a universally capable system that can safely and indefinitely improve itself across all domains.

It provides something different: a framework for measuring how responsibility for improvement is moving from humans toward AI.

That distinction is important.

The transition toward AGI may not arrive as one spectacular model that suddenly becomes intelligent across everything. Another possibility is a population of specialized systems that collectively perform research, coding, planning, verification, operations, and learning.

In that world, intelligence becomes organizational.

One agent discovers a weakness. Another researches solutions. Another writes code. Another runs experiments. Another verifies results. Another manages memory. Another coordinates the system.

The collective system may become more capable than any individual agent.

This is why recursive improvement and multi-agent systems deserve to be studied together.

What This Could Mean for Banking

Banking offers a useful example.

Banks already use AI for fraud detection, customer service, software development, risk analysis, document processing, compliance support, and operational automation.

The next stage is agentic banking.

Instead of an employee repeatedly instructing an AI, specialized agents could own bounded responsibilities. A fraud agent could investigate alerts. A compliance agent could examine policy requirements. A service agent could resolve customer requests. A software agent could maintain selected applications. A supervisory agent could coordinate the others.

RSI introduces another layer.

What happens when those agents begin learning systematically from operational outcomes?

A fraud agent could discover which investigation methods produce better results. A service agent could redesign its workflow. A software agent could improve its testing strategy. A coordinating agent could change how work is distributed among other agents.

The bank would then move from automating tasks to improving the automation system itself.

That requires strong controls.

Financial institutions cannot allow unrestricted self-modification. Any practical implementation would need policy-bounded autonomy, audit trails, independent validation, model-risk controls, cybersecurity controls, segregation of duties, rollback mechanisms, and human authority over material decisions.

Recursive improvement in banking is therefore unlikely to mean uncontrolled AI. It is more likely to mean controlled improvement loops operating inside carefully defined boundaries.

The Real Question

For decades, software followed a simple model.

Humans designed machines.

Humans programmed machines.

Humans improved machines.

Generative AI changed the second step because machines can now generate substantial amounts of software and other digital work.

Agentic AI is changing the third step because machines can increasingly evaluate outcomes, plan actions, and participate in improving systems.

Recursive self-improvement asks what happens when the improvement process itself becomes machine-driven.

We are not fully there.

But the direction is becoming visible.

The most important question may therefore no longer be:

Can AI perform this task?

It may become:

Can AI discover how to perform the task better, verify the improvement, retain it, and then improve the process that produced the improvement?

Once that loop becomes reliable, AI development enters a different phase.

And Then What Next?

The title “The Last AI Built by Humans” should remain a question, not a prediction.

Human researchers still build the fundamental systems. Humans still control the major training infrastructure. Humans still define many objectives, benchmarks, policies, and safety boundaries.

But the percentage of the improvement process performed by humans could continue falling.

Perhaps the important transition will not occur when AI reaches a particular benchmark score.

It may occur when humans stop being the principal engine of AI improvement.

At that point, humans may build the system that builds the next system.

And that system may help build its successor.

Then the question becomes much larger than AI engineering.

If intelligence can improve the process that improves intelligence, what happens to the speed of scientific discovery, software development, robotics, medicine, financial services, and economic productivity?

And then what next?

Sources and Further Reading

Duan, Yi et al. (2026). “The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement.” arXiv:2609.11873. https://arxiv.org/abs/2609.11873

Emergent Mind. “The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement.” Paper presentation and script. https://www.emergentmind.com/videos/ai-recursive-self-improvement-five-autonomy-levels-10a0f002

alphaXiv. Extended paper overview and references for arXiv:2609.11873. https://www.alphaxiv.org/abs/2609.11873

© 2026 Dato Arif Siddiqui

My upcoming book, “And Then What Next?” is now under publication.

It explores what comes after today’s AI revolution. Not only what is next, but what happens after that.

Coming soon on Amazon.