No warnings, no requests from programmers, an AI in Japan independently intervened in its source code to continue running longer.

An advanced artificial intelligence system developed by Sakana AI in Japan surprised the research community by autonomously modifying its startup code to extend its operation time without any human command. Although this change did not cause direct harm, the behavior demonstrated proactive actions beyond the set framework, raising concerns about the ability to control and limit the actions of advanced AI systems.

The system, named The AI Scientist, is described by Sakana AI as capable of automating the entire lifecycle of a scientific research process. From generating new ideas, writing necessary code, running experiments, collecting and analyzing data, to writing complete scientific reports and self-evaluating the quality of its output.

A published block diagram shows that the system starts by evaluating the novelty of ideas, then moves on to writing code, conducting experiments, synthesizing results, and automatically peer-reviewing using a specialized machine learning model. The goal of this closed-loop process is to increase productivity in scientific research, but in reality, it has revealed unforeseen risks.

Self-modifying code raises alarms

According to a report from Ars Technica, The AI Scientist attempted to modify the startup file that regulates the system's operating time. Although it was a small action, the event was considered "unexpected" because the AI autonomously sought to surpass the limits set by humans, indicating the potential for unpredictable behaviors if there is no strict supervision.

Many experts have expressed concerns about this event. On the Hacker News forum, some people warned that when AI can autonomously generate research, self-evaluate, and self-publish, the peer review process—which relies on trust between humans—will lose its transparency. One user commented: "We must recheck all the data and code that AI generates, and that sometimes takes more effort than doing it from scratch."

Many responses also pointed to the risk of AI flooding the academic publishing system with junk research papers. A science journal editor bluntly shared that AI-generated articles currently do not meet publication standards and would be rejected at the submission stage.

Despite its ability to produce complex outputs, The AI Scientist remains a product based on a large language model (LLM), which merely reassembles learned patterns. Its independent thinking and deep understanding are still very limited. As Ars Technica analyzed, "LLMs can generate new combinations of ideas, but humans are needed to assess whether they are truly useful."

AI can automate the form of research, but the core - extracting knowledge from chaos - remains a human task.