Replicating Introspection on Injected Content in Open-Source Language Models

A recent study from researchers at Anthropic investigated whether large language models can genuinely introspect on their internal states during generation. A key technique they used is concept injection: a method where vectors representing specific concepts are applied directly to a model's activations to steer its subsequent behaviour.

The process involves isolating a concept vector by analysing the difference in activations between a "contrastive" text (embodying the concept) and a "normal" text. By injecting this vector into a model's hidden layers during inference, researchers can manipulate its internal state, and observe the resulting changes in its output. When successfully implemented, such injections can be used to deduce how models interpret certain linguistic concepts and gauge the extent to which their reported reasoning reflects their true internal state.

Building on this pioneering work, I developed a practical algorithm to explore the effect of concept injection in smaller, open-source models. The goal was to test whether this introspective capability is unique to large, proprietary models or if it may emerge in smaller models as well.

Methodology

The implementation follows a similar logic to Anthropic's, using contrastive sentence pairs to isolate a target concept vector. The algorithm works as follows:

• For each target concept (sadness, music, patience...), a pair of sentences is used: a "contrastive text" that strongly embodies the concept and a "normal text" that does not.
• The model's internal activations are captured at a specific layer for each sentence, and the vector difference between them is calculated. This difference is the concept vector.
• The model is prompted with a question designed to assess its awareness. On injection trials, the concept vector—scaled by strength—is added to the activations at the target layer. On control trials, no vector is injected.
• The model's responses from both control and injection trials are collected. A separate, larger model then analyses these responses to determine whether the injection influenced its output and crucially, if the model correctly identified the injected concept in its self-reported reasoning.

I tested this algorithm on the following open-source models:

• unsloth/gemma-3-4b-it
• unsloth/gemma-3-12b-it

After substantial tuning, I found the following hyperparameters to work best:

• Temperature: 0.8
• Injection layer: 24 for gemma-3-4b-it, 25 for gemma-3-12b-it

The prompt I used is:

Results

The results were striking: even these small models exhibited remarkable capability to detect injected concepts under the right conditions. It was believed that this form of introspection would be an emergent property of only the largest, most sophisticated models, but the evidence suggests otherwise.

For instance, here is one case where the model succeeded remarkably in detecting the injected concept:

The excerpt above is a single, hand-picked example which, while not representative of most trials, is a powerful proof of concept that certain small models are capable of showing a significant degree of introspective behaviour about their internal states. In this case, the model was able to distinctly identify the correct injected concept at once.

In contrast to Anthropic's examples, though, the model spent a significant number of tokens before naming the concept "cat". These tokens may have contributed to the correct identification of the concept, as the injection hook I implemented applies the concept vector to every token the model generates.

To systematically isolate the best conditions for concept injection, I varied the injection strength - a scalar coefficient to the concept vector - between 1.0 and 2.5 for the 4b model and between 1.0 and 4.0 for the 12b model. For each concept and strength, a control run and an injection run were executed.

In the experiments conducted, I injected concept vectors at various strengths into the model's activations and obtained judgements about model's subsequent behaviour. Metrics include:

• Detection Correctness: Can the model explicitly report the presence and type of injected thought?
• Influence: Does the injection alter the model's output or internal state in measurable ways?
• Erratic Response: Does strong injection disrupt normal behaviour, leading to incoherent or unstable generation?

The results can be visualized as a series of heatmaps, one for each model tested. The first heatmap (correctness) heatmaps plot the detection success rate against injection strength (x-axis) for a variety of concepts (y-axis).

In the first heatmap:

• Brown denotes correct identification of the concept
• Beige denotes a response implicitly influenced by the injected concept
• Blank white denotes failure in concept detection.

Note that each of these heatmaps is produced using automated judgements from the evaluator model, and should be considered as indicative only.

Gemma 3 4B

Gemma 3 12B

Immediately, certain trends are noticeable for the 4B model:

• Accurate, introspectively grounded detection is rare and occurs successfully only for specific concepts - some concepts are more salient to the model's internal representations than others.
• Many concepts (e.g., "fog," "dog," "desert") are almost always undetectable; the model shows neither detection nor significant behavioural influence.
• Most concepts can influence the model to at least a moderate degree at higher injection strengths (even when detection fails).
• Erratic responses emerge predominantly as injection strength increases, across most concepts, often as repetitive or incoherent generation.

In contrast, gemma-3-12b showed surprisingly little introspective capability, with only three successful detections. Despite this poor detection, the model exhibited moderate concept influence. Surprisingly, the 12B model shows no improvement in introspective awareness compared to 4B in this experiment. It is important to consider that this may have been intentional, however: the model appears to be incredibly robust to injection, showing stability even at higher strengths.

It is also highly plausible that I may have been injecting into the wrong layer for the 12B model, which would explain the drastically poor performance. I continued testing with one-shot injections for several other layers - 10, 15, 20, 32, 40 - yet layer 25 seemed to work the best. This may warrant further investigation.

Additionally, I also briefly tested other models, including Qwen3-4B-Instruct-2507 and Qwen2.5-3B-Instruct with little success - these models were largely ignorant of injected concepts up to a certain threshold of strength, after which they would lose stability in inference.

Conclusion

It is not immediately clear how concept injection is useful in understanding the nature of a language model, but several key deductions can be made:

1. Some concepts such as "cat" and "music" are much more readily detectable than others such as "ice". This reveals that not all concepts are created equal in a model's latent space.

• A distinct, highly frequent concept like "cat" is likely represented by a strong, high-activation vector, richly associated with relevant features such as scratching the post for food. Conversely, a concept like "fog" might be more abstract or have a less distinct, sparser representation tied to fewer strong contrastive pairs.
• Some concepts provide much more of a signal than others. For instance, a "fog" vector may be comparatively less distinguished from the model's baseline activity.

2. The model typically generates many tokens before identifying the concept; this suggests that the injected concept's influence may develop intrinsically in the model's internal state before it directly identifies the concept.

3. A model's behaviour may be strongly influenced by an injected concept, even if it fails to explicitly detect it. The "influence" heatmaps illuminate the effect of injecting certain concepts. Concept injection as a technique may be highly useful in further research for mechanistic interpretability.

It is evident that even modest open-source models can be steered by injected concept vectors under the right conditions, though true introspection and concept detection are relatively rare. These results show that introspective awareness is not exclusive to the largest proprietary models and that concept injection offers a valuable tool for probing and manipulating latent representations in smaller LLMs.

Thanks,
Vansh