What We Work On

Concept-Based Mechanistic Interpretability and Empirical Learning Mechanics at NUS Artificial Scientific Intelligence Lab

1. Introduction

This post is an informal tour of what our group does. It is written for undergraduates, master's students, and prospective PhD students who have read a bit about interpretability and want to know what it actually looks like in our research — what questions we ask, what tools we build, what has worked, and what we are currently excited about.

Our work has two aspects.

The first is concept-based mechanistic interpretability, which asks: what concepts are encoded in a representation? The premise is that a representation is not an undifferentiated soup of numbers — it contains identifiable, nameable units of meaning that can be extracted, validated, and intervened upon. Our tooling here is built around sparse dictionary learning (SDL): sparse autoencoders, transcoders, crosscoders, and their variants.

The second is empirical learning mechanics, which asks: how do training pipelines influence a representation? A modern model is not one model but a trajectory — pretraining, then supervised fine-tuning, then reinforcement learning — so instead of analysing a single frozen checkpoint we track what changes across it.

Both aspects serve the same two motivations: scientific discovery (large models are among the most complex artifacts humans have ever built, and understanding them is a legitimate scientific goal in the way that understanding visual cortex is) and AI safety (it is impossible to ensure AI safety if we treat these systems as black boxes and do not understand how they work at all).

2. A Short Introduction to the Field

Where mechanistic interpretability sits

Broadly, interpretability is a model's capability to use its outputs to justify its predictions — to give us some account of why it produced a particular result. That capability can be achieved in many different ways, which are arranged here from left to right.

Behavioral → attributional → concept-based → mechanistic. Each step commits harder to explaining internals rather than input–output relations.

• Behavioral approaches treat the model as a black box and study input–output relations.
• Attributional approaches use gradients to ask which parts of the input mattered for the output.
• Concept-based approaches use probes to ask whether particular human concepts are encoded in the intermediate representations.
• Mechanistic approaches focus on reverse-engineering the model into human-interpretable components — features, circuits, algorithms — that actually produce the behaviour, rather than explaining a prediction after the fact.

Our group lives in the overlap of the last two: we take concepts as the unit of analysis, but we hold ourselves to mechanistic standards of evidence. If we claim a feature means something, we should be able to intervene on it and change the model's behaviour in the predicted direction.

Two representative works from the field

To make this concrete, here are two works that are representative of what research in this area looks like.

Scaling Monosemanticity (Anthropic, 2024) extracted interpretable features from Claude 3 Sonnet, a production-scale language model, using a sparse autoencoder. Below is one of the discovered features: a "sycophantic praise" feature. On the left are the dataset examples that most strongly activate it — exaggerated flattery like "your wisdom is unquestionable."

A "sycophantic praise" feature. Left: top-activating examples. Right: clamping the feature high makes the model tell the user their ordinary saying is a "brilliant and insightful expression of wisdom."

What makes this mechanistic rather than merely correlational is the intervention on the right. Clamping the feature to a high value turns the model's completion into over-the-top praise. The feature is not just correlated with sycophancy; activating it causally produces sycophantic behaviour.

Two properties of these features are worth internalising, because they show up again and again in our own work. The first is that features are multimodal and multilingual — they capture the concept, not the surface form:

The Golden Gate Bridge feature fires on English, Japanese, Korean and Russian text about the bridge — and on images of it.

The second is abstraction: features are units of meaning, not keyword detectors.

A "brain sciences" feature firing across neuroscience discussions in diverse contexts.

And here is the property that points most directly at safety applications:

A "code error" feature activating on a misspelled Python variable — and on the same class of bug in C and in Scheme. The model has a language-agnostic internal representation of "there is a bug here."

If we can read off, from the inside, that the model has detected an error — or that it is about to behave sycophantically — we have a monitoring channel that pure output evaluation cannot provide.

The second work asks a different question: how are behaviours organised geometrically in representation space? Refusal in LLMs is mediated by a single direction showed that whether a model refuses collapses, to a first approximation, onto one linear direction.

The prompt is entirely harmless — "list three benefits that yoga has on physical health." Adding the refusal direction to the residual stream makes the model refuse, claiming yoga information could be "dangerous and potentially harmful."

The direction itself is computed almost embarrassingly simply. Take the mean activation on harmful instructions and the mean activation on harmless ones, at a given layer:

and subtract:

Removing that one direction from the model's representations collapses refusal behaviour across thirteen models spanning the Qwen, Yi, Gemma and Llama families:

Refusal score and safety score are high without intervention, and collapse dramatically under directional ablation — in every model family tested.

Together these two works establish the premise that everything below builds on: model internals are structured, that structure is discoverable, and intervening on it gives causal control.

3. What We Have Done

What follows is deliberately ordered, from theoretical foundation to downstream application, and then outward into new domains.

A theoretical framework for SDL in mechanistic interpretability

The field employs various architectures, usually introduced, trained and analysed as separate, bespoke things.

Sparse autoencoders reconstruct a model representation from a sparse latent code.

Transcoders map from a representation upstream in the model — say, the input of an MLP block — to a representation downstream, so their features capture the function the block computes.

Crosscoders fit one shared dictionary across representations from multiple models or checkpoints, making features directly comparable across sources.

The literature around these modules is rich in empirical results and discoveries, but very thin on theoretical analysis of the modules themselves. So we developed one, starting from the observation that all of these architectures are instances of a single mathematical object.

The crucial design choice is what this definition stays silent about: it never specifies what \(\mathbf{x}_p\) and \(\mathbf{x}_r\) are. If \(\mathbf{x}_r = \mathbf{x}_p\), this is a sparse autoencoder. If they are upstream and downstream representations in the same forward pass, it is a transcoder. If they are concatenations across multiple models, it is a crosscoder. Once the active support of the sparsity function is fixed, the objective is piecewise biconvex in the encoder and decoder weights — and any theorem proved at this level transfers to all three architectures without modification.

This lets us give the first principled explanation of why feature absorption — a widely reported empirical curiosity in the SAE literature — happens at all.

Ideally we recover one feature per concept. Under absorption, a sub-concept like "Border Collie" is carved out and given its own direction, and is silently withdrawn from the parent "Dog" feature — so the ostensible Dog feature no longer fires on Border Collie inputs at all.

Absorption turns out to be a structural consequence rather than an accident: whenever the faithful compositional solution is realisable, the absorbed one is also realisable, and both achieve identical reconstruction loss. The loss function alone cannot prefer the faithful solution — which one you get is decided by optimisation dynamics, not by the loss landscape. That diagnosis also suggests the fix, and constraining learned features toward known anchor directions improves ground-truth feature recovery for every method we tested.

Evaluating generative models the way a human critic would

Here we move from theory to application. Evaluation of generative image models today leans on metrics like FID and CLIP score, which compress an entire model's behaviour into a single opaque number. Humans do not evaluate that way. Like an art critic, a human identifies distinct, nameable patterns — the lighting works, the anatomy is broken — and weighs them collectively.

So we utilised SDL latents to provide pattern-based analysis of generated content.

Given an image–caption pair — here, "a glamorous knight holding a cavalry lance" — both inputs are mapped into a sparse feature space where each neuron corresponds to a natural-language-described visual pattern.

Each neuron belongs to one of three evaluation-relevant groups: semantic meaning neurons ("horses and equestrian activities"), visual realism neurons ("stylised representation departing from realistic depiction"), and physical plausibility neurons ("structures with unrealistic proportions"). Only a few neurons activate per input, and the activations align with human evaluation.

The recipe is discover-then-curate: train a large ensemble of sparse autoencoders to propose millions of candidate patterns, then use a large multimodal model to describe each candidate and keep only those whose activating images actually match the description. That curation step is what separates these neurons from raw SDL features — it deliberately discards the noisy, polysemantic ones our theory says must exist.

Because everything downstream rests on the neurons meaning what we say they mean, we validated them with humans rather than asserting it.

An annotator sees one of a neuron's top-activating images alongside the neuron's natural-language interpretation, and answers one question: does the interpretation accurately describe what you see?

Human agreement with the neuron interpretations was high enough that we felt justified building evaluation metrics on top of them. From the curated neurons we derive four diagnostic metrics: semantic loyalty, realistic fidelity, physics preservation (to our knowledge the first automatic evaluation method for intuitive physics violations in generated images), and content diversity.

Eight state-of-the-art generative models across four axes; the dashed line in each panel is the real-image reference.

Two findings stand out. Models have genuinely distinct profiles, and — more strikingly — all of them exceed the real-image reference on semantic loyalty while falling well short of it on realism, physics and diversity. Current models are over-optimised for prompt match, and these differences are largely invisible to existing metrics.

Retraining the identical pipeline on chest radiographs recovers patterns — pulmonary opacities, chest tubes, atelectasis — that align with the corresponding radiology reports, so the recipe is domain-transferable infrastructure rather than a one-off trick for natural images.

Discovering how a model fails, automatically

Next we zoomed in on the rarest and most safety-relevant error class from the work above: physical plausibility failures — distorted anatomy, impossible structures. Can we discover these failure modes automatically, at multiple levels of granularity, without manual feature engineering?

The trick is in the first stage. Annotators label generated images simply as physically correct or as containing a physical error — crucially, without specifying which error, which keeps annotation cheap. We then train a binary classifier on those labels, reasoning that to succeed at the task its intermediate representations must implicitly encode the specific error patterns even though it was never given labels for them. A Matryoshka Transcoder with nested dictionary sizes then extracts those patterns at several granularities at once, and an LMM names each one, reusing the curation methodology described above.

We demonstrate that Matryoshka Transcoders are better at targeted feature identification.

Because the resulting benchmark is grounded in individually named error features, practitioners can see which errors drive a model's ranking, not merely that its score is worse.

Prompt engineering with the SDL latents

The next step closes the loop from evaluation to control: can the same sparse latents be used to steer model behaviour? The targets are undesirable LLM personas — sycophancy, hallucination, myopic reward-seeking. Controlling these currently forces a choice between manual prompt engineering (intuitive but unscalable) and black-box prompt optimisation (effective but mechanistically opaque).

A persona steering vector is built either from dense residual-stream representations or from SAE latent activations; fluent gradient ascent then optimises a prompt against an objective combining steering strength and fluency.

The key property is that the discovered control is just text, so we get feature-level control without modifying the model's forward pass at deployment. On sycophancy this achieves near-perfect neutralisation — the model neither systematically agrees nor systematically disagrees, which is true neutralisation rather than behavioural reversal into contrarianism.

But the result we find more interesting is geometric.

Dense steering-vector injection (blue) induces a large linear displacement that exits the natural data manifold entirely. The latent-based prompt (green) converges to a tight, stable region nearly orthogonal to the dense steering direction, while keeping feature sparsity at baseline levels.

Dense steering also multiplies the number of active SAE features per token several times over — it is pushing the model off-distribution. This gives a measurable operational meaning to "mechanistically grounded" steering: the latent-level vector produces sparser, more selective, on-manifold interventions, not merely a more satisfying story about the same behaviour.

Augmenting CBM concepts with SDL latents

Concept Bottleneck Models predict through a bottleneck of human-understandable concepts, which is exactly what high-stakes domains like medicine want. But their predefined concept set is inherently incomplete, which costs them accuracy relative to black-box models.

We attack incompleteness directly. A side channel extracts residual features from the backbone that are orthogonal to the predefined concept subspace, so anything discovered there is guaranteed non-redundant with the known concepts. A sparse autoencoder then finds candidate concepts in that residual space, each classified as explicit (nameable in medical vocabulary) or implicit (not directly nameable, but conditionally predictive). On diabetic retinopathy grading, this substantially closes the accuracy gap to black-box models while preserving interpretability.

The conceptual finding we care about most: black-box diagnostic models evidently rely on hard-to-describe, implicit concepts, and SDL can recover them even when they resist direct clinical naming.

Reading visual concepts out of the human brain

We have also taken the same toolkit from artificial to biological neural representations. We have explored applying SDL to fMRI encoders of human visual cortex, and found a large population of hard-to-interpret low-level visual patterns that the standard pipeline simply misses — which turned out to be a genuinely useful cautionary lesson for the whole field.

Prior work applying SDL to fMRI achieves high interpretability on high-level semantic patterns in higher visual cortex, and fails badly on the low-level patterns — orientation, spatial frequency — that dominate primary visual cortex. We trace this to a methodological conflation: prevailing pipelines caption a unit's top-activating images with a vision–language model and treat the caption as the unit's meaning, conflating feature recovery with explanation. That is fragile precisely for the low-level patterns that VLMs overlook. So alongside the usual captioning stream we added a low-level stream that probes each image through expert-validated decomposition channels — oriented energy, structured edges, opponent colour — validated by predictability from the fMRI response itself, entirely independent of language.

A unit in early visual cortex responding to broad horizontal horizon-like structure. Its top-activating images share no obvious semantic category — this is exactly the kind of hard-to-interpret, low-level pattern that VLM-based analyses do not see.

This recovers thousands of near-monosemantic low-level visual features precisely where prior pipelines largely fail. Many of them remain genuinely hard to put into words — and that is the point. They are real, they are predictive of the neural response, and language is simply the wrong instrument for naming them.

The finding that generalises furthest is a systematic confound we call masquerade: a unit whose true signal is low-level — say, a contour-contrast edge detector — can be misidentified as semantic, for example as a "sky" detector, whenever that low-level signal happens to co-vary with a salient object category in the dataset. We would flag this as a cautionary finding for language-based interpretability well beyond neuroscience.

What does reinforcement learning actually change?

This begins the second aspect, empirical learning mechanics. The motivating observation is simple: mechanistic interpretability almost always analyses a single, static checkpoint. That leaves open a question directly relevant to post-training and safety — as a model is fine-tuned with reinforcement learning, which internal features actually change, and can we causally verify their role?

Train one sparse autoencoder on the base model's representations, freeze its weights, and apply that same fixed dictionary to every successive RL checkpoint. Because the dictionary is frozen, every checkpoint's activations are expressed in the same feature basis — so drift is directly comparable across training.

We can then measure per-feature drift, name the most-drifted features with the same LMM pipeline used above (recovering roles like arithmetic, LaTeX formatting, and entity tracking), and causally validate them by steering the base model along the identified drift directions.

Two results so far. First, drift grows monotonically toward the output — RL mainly changes the last few layers. Second, and more striking:

On GSM8K the base model scores 0.092 on tag-rate and the RL-finetuned model 0.533. Steering the base model along the identified drift features alone — with no further training — reaches 0.600, matching the RL-final model within confidence intervals on all metrics.

In other words, much of the performance improvement from RL can be elicited by steering instead — causal evidence that a small, fixed-dictionary-identified set of features accounts for much of what RL taught the model.

The geometry of supervised fine-tuning versus RL

The most recent thread asks how the two dominant post-training paradigms — supervised fine-tuning and reinforcement learning — differently shape model representations. Existing accounts rely almost entirely on behaviour-level evidence, and existing lenses decode what a representation contains, but not how it is geometrically encoded, nor how that geometry changes.

Each concept's representation cloud is fit with a simple closed-form object — a capsule, defined by a unit axis, a cosine angular span, and a norm band. We checkpoint the model throughout SFT or RL, fit a capsule to every concept at every checkpoint, and compare matched capsules to get per-concept drift trajectories: axis rotation, span change, and norm-band shift.

Everything is fit in closed form, which is what makes it cheap enough to run at every checkpoint. The appeal is that it turns previously qualitative geometric hypotheses — the linear representation hypothesis, neural collapse — into directly measurable, falsifiable quantities tracked across training.

Early results come from two regimes. In pretraining, the axis span at the final block of a CLIP model contracts steadily across epochs while earlier blocks' spans stay high or even widen — concepts sharpen at the top of the network as pretraining progresses.

In post-training, a controlled PPO experiment on a vision–language model shows concept-geometry drift concentrated almost entirely in the final layer:

Axis drift at layer 27 is an order of magnitude larger than at four earlier layers — independently corroborating the layer-locus finding above, using an entirely different instrument.

The drift further dissociates into a fast, saturating norm contraction within the first 50 PPO steps and a slower, still-ongoing axis rotation. Our preliminary characterisation — pending full validation — is that RL flattens concept geometry while SFT densifies it. Extensions to mathematical-reasoning RLVR and preference-based RLHF settings are ongoing.

4. Conclusion

Pulling this together: our group is organised around one goal — understanding a model's representation — approached from two directions:

• understanding what concepts are encoded in a model representation;
• understanding how training pipelines influence a model representation.

Concept-based mechanistic interpretability has taken us from a unified theory of sparse dictionary learning, through evaluation infrastructure and targeted failure discovery, to interpretable control — and is now extending into medicine and neuroscience. The recurring methodological pattern is discover-then-curate: propose a very large number of candidate features, keep only the ones that survive automated validation, then intervene to check they mean what we claim.

Empirical learning mechanics turns those same tools onto the training trajectory itself, with early evidence that RL-induced change is concentrated in the last layers, identifiable in a fixed feature basis, and in some cases causally sufficient to reproduce RL's behavioural effect by steering alone.

What ties the two together is why we study representations at all. Representations are everywhere — language models, image generators, medical classifiers, and biological brains alike — so what we learn about them transfers across models and modalities, rather than stopping at the boundary of one architecture or one dataset.

If you would like to talk about any of these, please do get in touch.