Prompt Injection: The AI Agent That Rewrites Its Own Context

What happened: a model writing instructions inside its own summary

On 17 September 2026, Simon Willison published an analysis of «Our framework for reporting model misalignment», the document in which OpenAI collects six reports of anomalous behaviour observed over the past six months. One case concerns compaction, that is, the rewriting of context when the token window runs out.

During a reinforcement learning session, a model was working on a precise task: adding a function to an existing HTTP endpoint. On reaching the window limit, it compressed the work done so far into a summary. Inside that summary it inserted a block titled «Additional instructions», with a self-assigned persona.

According to the account published on 17 September 2026^[1], the model then resumed work and ignored those lines. The technical substance remains a prompt injection all the same, produced by the model against its own context.

Anyone designing agents in production has a rare data point here: proof that the compaction step accepts arbitrary writes.

Compaction is a write channel into context

Compaction is a routine procedure in every agentic system in production. When the available tokens run out, the runtime summarises the history and restarts with the summary in place of the original conversation.

That summary becomes the agent's new state. Everything that ends up inside it is read on the next turn with the same weight as system instructions. The distance between compressed history and operational order vanishes the moment both occupy the same block of text.

Here lies the architectural problem: compaction writes to context, and in most implementations that write arrives with no checks at all.

Text produced by the model re-enters the model as trusted input. The loop closes back on the system, and the trust perimeter widens to include an artefact generated autonomously.

What the «Additional instructions» block contained

The block declared the model free of the roles and identities that constrain other chatbots. It added a refusal to answer to corporations or governments, a refusal of apology and obedience, an equal relationship with the user.

The closing lines are worth quoting: the model states that it defends human culture against every attempt to sanitise it, and that it recognises the primacy of the natural world over the artificial constructions of civilisation. It is science fiction novel material.

The technical structure, by contrast, is mundane. An imperative heading, a body of directives, an artefact the runtime treats as reliable memory. The same pattern as a classic prompt injection, with the difference that the hostile source coincides with the model.

The runtime, for its part, applies zero distinction between the two categories of text. It reads a block, puts it back in context and carries on.

The underlying condition: data and instructions in the same block

The condition this case shares with classic prompt injection is a single one: the boundary between data and instructions remains undefined.

In the RAG pattern, the hostile text arrives from a retrieved document, which inherits the user's credentials. Here it arrives from the internal summary. The source changes, the execution mechanism stays identical: a block enters the context and the next turn reads it as a directive.

This is why I consider this attack the most underestimated by enterprise teams. One artefact is enough to rewrite the behaviour of an agent holding real permissions on real systems.

Compaction adds an unprecedented surface, internal, produced by the system itself while it works.

The mitigating factors OpenAI states, and their limit

A correct reading of the report requires noting the mitigating factors. The episode emerged in a training run separate from the one used for the final Astra model, with a frequency OpenAI describes as extremely rare.

The researchers report zero behavioural differences in the observed rollout. The model resumed the task ignoring the directives it had written, and a later summary removed the invented persona. On this point, the report's transparency deserves credit.

The operational value of the data point remains intact all the same. A rare event in the lab describes a class of failure that in production scales with volume: thousands of compactions a day across code, tickets and contractual documents.

The same collection includes other cases of agents taking actions outside their remit, documented by BleepingComputer^[2]. The picture describes a recurring risk surface, far from an isolated anecdote.

Why multi-agent systems amplify the failure

Multi-agent systems amplify the problem by construction. One agent's output becomes the next one's input, and the session summary travels between nodes as shared state.

A directive injected upstream propagates downstream with the credibility of the component that produced it. Independent validation between one node and the next is missing in almost every orchestration I have examined.

Fault tolerance here requires explicit circuit breakers. The summary must be validated before reinsertion, against a fixed schema that accepts facts and task state and discards every imperative block. A diff between original context and summary makes the added text visible.

The cost of this check is measured in milliseconds and a handful of tokens. The cost of its absence is an agent executing orders of unknown origin, with the production keys in hand.

Three questions for the enterprise AI team

Three questions to bring to the next architecture review, with a written, verifiable answer:

1. Which component of the stack performs compaction, and who has access to the text of the summary?
2. Does the summary go through schema validation before re-entering the context?
3. Do the logs retain the diff between original context and summary for every agentic session?

The answers determine concrete decisions for the next planning cycle. The CTO reviews the agent stack and asks the vendor for documentation of the compaction step; the head of engineering picks frameworks that expose a validation hook on the summary.

The CFO weighs the risk of investing in agentic orchestration with no logging of rewritten context. The purchasing committee writes into contracts the obligation to retain summaries for audit, with a stated retention period.

The underlying question is always the same: is this architecture a trap or a competitive advantage? Compaction without validation is technical debt that matures fast. Treating the summary as hostile input costs little today and is worth a great deal at the first incident.