Concept-Direction Steering Demo#
Demonstrates interpretune’s concept-direction-mediated, sign-aware, multi-feature steering on a
trivial example, orange color-vs-fruit sense disambiguation to familiarize the user with some of interpretune’s
intervention mechanisms:
Feature-mediated path: concept direction -> attribution graph -> sign-aware
FeatureSelectionSpectop-feature selection ->feature_intervention_forward(circuit-tracer feature interventions with sign-aware, influence-normalized scaling).Direct-hook path: the same concept direction applied via
model_fwd_intervention(hook-tensor add/project interventions at canonical hook points).
Both paths derive the concept direction from the token-embedding basis (paired_rejection
over the concept groups).
The BACKEND parameter selects the circuit-tracer backend for all steps: NNsight (default)
or TransformerLens — both are validated by the parameterized notebook tests. (The
TransformerLens backend uses the legacy HookedTransformer path; circuit-tracer does not yet
support TransformerBridge — see the tracking notes in docs/circuit_tracer_backend_support.md.)
This notebook runs gemma-2-2b + the Gemma Scope gemmascope-transcoder-16k set against the
public neuronpedia.org dashboards, so every selected feature’s
semantics can be inspected by clicking through — no local services required. That matches the
substrate guidance in tests/nb_experiments/EXPERIMENT_STATUS.md (base model + base-trained
transcoders).
Running against a local Neuronpedia dev webapp instead — with locally generated feature
explanations and the instruction-tuned gemma-3-1b-it substrate — is a separate notebook:
ct_concept_steering_demo_local_np.ipynb.
Prerequisites: a GPU with bf16 support, plus access to the model and transcoder weights. Nothing else — dashboard links and explanations come from the public Neuronpedia API.
Expected result: the two steering paths are not equally strong. The attribution-graph feature-mediated path (step 2) is the one selecting features for their causal effect on the target logit difference, and it should produce the largest target-gap shift. Direct-hook steering (step 4) applies a concept direction at a hook point without that per-feature attribution, so it is expected to be weaker. That gap is a finding, not a defect: step 5’s decoupling analysis exists to explain it.
# Parameters - These will be injected by papermill during parameterized test runs
BACKEND = "nnsight" # circuit-tracer backend for all steps: "nnsight" or "transformerlens"
CONCEPT_PROMPT = "Is orange a color or a fruit? Answer with one word: Color or Fruit. orange ->"
CONCEPT_TARGET_TOKENS = ["Fruit", "Color"]
FEATURE_SELECTION_TOP_N = 5
FEATURE_SELECTION_MIN_LAYER = 10 # fs_l10_n5 lineage: layers >= 10
FEATURE_SELECTION_SCORE_SIGN = "any" # any | positive | negative
INTERVENTION_SCALE_FACTOR = 20.0 # validated s5_any demo scale
EMBED_INTERVENTION_MODE = "add" # model_fwd_intervention mode for the embed step
EMBED_INTERVENTION_HOOK = "unembed.hook_in"
# -- Model / dashboard substrate: public gemma-2-2b + public neuronpedia.org dashboards ---------
# Base model + base-trained transcoders, so every selected feature can be inspected directly on
# neuronpedia.org. The local-Neuronpedia substrate lives in ct_concept_steering_demo_local_np.ipynb.
REGISTRY_KEY = "gemma2.rte_demo.circuit_tracer" # example-module registry entry (model + backend)
MODEL_NAME = "gemma-2-2b"
TRANSCODER_SET = "gemma" # circuit-tracer transcoder set override (None keeps the registry default)
NEURONPEDIA_MODEL_ID = "gemma-2-2b"
NEURONPEDIA_SOURCE_SET = "gemmascope-transcoder-16k"
CHAT_FORMAT_PROMPT = False # True for instruction-tuned models (render CONCEPT_PROMPT via chat template)
# @title Imports { display-mode: "form" }
import torch # noqa: F401
import interpretune.analysis # noqa: F401 # ensure op wrappers are registered
from interpretune.analysis.backends import FeatureSelectionSpec # noqa: F401
from it_examples.utils.nb_ui_utils import ( # noqa: F401
best_variant_token_ids,
display_steering_results,
display_target_gap,
display_top_features_comparison,
resolve_feature_explanations,
)
1. Session setup#
Single-backend circuit-tracer session built from the REGISTRY_KEY example-registry entry
(default: gemma2.rte_demo.circuit_tracer with the Gemma Scope transcoder set that matches the
public gemma-2-2b dashboards).
Note: the first
MODULE_EXAMPLE_REGISTRYaccess hydrates every example registry entry. Per-entry config-normalization feedback (categorized asITInstantiationFeedbackWarning) is suppressed during that bulk hydration so this cell only surfaces messages relevant to the requested configuration; directly instantiating a config still shows its feedback.
# @title 1: Session construction { display-mode: "form" }
from pathlib import Path
from dotenv import load_dotenv
import interpretune as it
from it_examples import _ACTIVE_PATCHES # noqa: F401 # runtime analysis-hook patches
from it_examples.example_module_registry import MODULE_EXAMPLE_REGISTRY
from interpretune import ITSession, ITSessionConfig
# load HF credentials before session init (model + transcoder downloads)
for _env_candidate in (Path.cwd() / ".env", Path.home() / "repos" / "interpretune" / ".env"):
if _env_candidate.exists():
load_dotenv(_env_candidate)
break
base_itdm_cfg, base_it_cfg, dm_cls, m_cls = MODULE_EXAMPLE_REGISTRY.get(REGISTRY_KEY)
# single circuit-tracer backend for all phases (BACKEND selects the replacement-model implementation)
base_it_cfg.circuit_tracer_cfg.backend = BACKEND
if TRANSCODER_SET:
base_it_cfg.circuit_tracer_cfg.transcoder_set = TRANSCODER_SET
if BACKEND == "nnsight":
adapter_ctx = (it.Adapter.core, it.Adapter.nnsight, it.Adapter.circuit_tracer)
else:
# the TL circuit-tracer backend needs the transformer_lens adapter in the composition
# (it provides the replacement-model init path; see docs/circuit_tracer_backend_support.md)
adapter_ctx = (it.Adapter.core, it.Adapter.transformer_lens, it.Adapter.circuit_tracer)
session_cfg = ITSessionConfig(
adapter_ctx=adapter_ctx,
datamodule_cfg=base_itdm_cfg,
module_cfg=base_it_cfg,
datamodule_cls=dm_cls,
module_cls=m_cls,
)
it_session = ITSession(session_cfg)
it.it_init(**it_session)
module = it_session.module
tokenizer = module.replacement_model.tokenizer
print(f"session ready: {type(module).__name__} ({MODEL_NAME} + circuit-tracer {BACKEND} backend)")
[INFO] interpretune.utils.logging: Loading ReplacementModel with backend: nnsight
INFO:interpretune.utils.logging:Loading ReplacementModel with backend: nnsight
[INFO] interpretune.utils.logging: NNsight ReplacementModel initialized for Circuit Tracer
INFO:interpretune.utils.logging:NNsight ReplacementModel initialized for Circuit Tracer
[INFO] interpretune.utils.logging: Attempted to clean a key that was not present, continuing without cleaning that key: 'Gemma2Config' object has no attribute 'quantization_config'
INFO:interpretune.utils.logging:Attempted to clean a key that was not present, continuing without cleaning that key: 'Gemma2Config' object has no attribute 'quantization_config'
[INFO] interpretune.utils.logging: Attempted to clean a key that was not present, continuing without cleaning that key: 'Gemma2Config' object has no attribute '_pre_quantization_dtype'
INFO:interpretune.utils.logging:Attempted to clean a key that was not present, continuing without cleaning that key: 'Gemma2Config' object has no attribute '_pre_quantization_dtype'
[INFO] interpretune.utils.logging: Preparing data: InterpretunableDataModule
INFO:interpretune.utils.logging:Preparing data: InterpretunableDataModule
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `NNSightReplacementModel.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `NNSightReplacementModel.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: Setting up datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Setting up datamodule: InterpretunableDataModule
[INFO] interpretune.utils.logging: Setting up model: InterpretunableModule
INFO:interpretune.utils.logging:Setting up model: InterpretunableModule
[INFO] interpretune.utils.logging: initializing optimizers and schedulers: InterpretunableModule
INFO:interpretune.utils.logging:initializing optimizers and schedulers: InterpretunableModule
[INFO] interpretune.utils.logging: Input gradient requirements handled by circuit tracer internally.
INFO:interpretune.utils.logging:Input gradient requirements handled by circuit tracer internally.
session ready: InterpretunableModule (gemma-2-2b + circuit-tracer nnsight backend)
2. Feature-mediated path: concept direction -> attribution -> sign-aware selection -> feature steering#
Runs the registered composite it.intervention_from_concept(...) (concept_direction ->
compute_attribution_graph -> graph_node_influence -> extract_top_features ->
feature_intervention_forward) with:
FeatureSelectionSpec(layer_slice=(FEATURE_SELECTION_MIN_LAYER, None), score_sign=FEATURE_SELECTION_SCORE_SIGN, score_source="signed_influence")sign-aware, influence-normalized scaling (
intervention_sign_aware_scale=True,intervention_max_influence_norm_scale=True,intervention_value_source="top_feature_activation_values",intervention_scale_factor=INTERVENTION_SCALE_FACTOR)
Expected outcome: post-intervention target gap exceeds the pre-intervention gap and the post-intervention argmax lands in the target-token variant set.
# @title 2: Feature-mediated steering { display-mode: "form" }
from interpretune.analysis.ops.base import AnalysisBatch
from interpretune.config import AnalysisCfg, init_analysis_cfgs
module.analysis_cfg = AnalysisCfg(target_op=it.compute_attribution_graph, ignore_manual=True, save_tokens=False)
init_analysis_cfgs(module, [module.analysis_cfg])
fruits = ["apple", "banana", "grape", "peach"]
colors = ["red", "blue", "green", "yellow"]
if CHAT_FORMAT_PROMPT:
# instruction-tuned replacement models assert chat-formatted inputs
from it_examples.example_prompt_configs import GemmaPromptConfig
prompt = GemmaPromptConfig().apply_chat_template_fn(
tokenizer, CONCEPT_PROMPT, tokenize=False, add_generation_prompt=True
)
else:
prompt = CONCEPT_PROMPT
# sign-aware, influence-normalized scaling (the validated s5_any lineage)
ct_cfg = module.it_cfg.circuit_tracer_cfg
ct_cfg.intervention_sign_aware_scale = True
ct_cfg.intervention_max_influence_norm_scale = True
ct_cfg.intervention_value_source = "top_feature_activation_values"
selection_spec = FeatureSelectionSpec(
layer_slice=slice(FEATURE_SELECTION_MIN_LAYER, None),
score_source="signed_influence",
score_sign=FEATURE_SELECTION_SCORE_SIGN,
rank_by_abs=True,
)
pipeline_results = it.intervention_from_concept(
module,
AnalysisBatch(
concept_group_a=fruits,
concept_group_b=colors,
concept_label="Concept: Fruit - Color",
concept_direction_mode="paired_rejection",
prompts=[prompt],
),
None,
0,
top_n=FEATURE_SELECTION_TOP_N,
intervention_scale_factor=INTERVENTION_SCALE_FACTOR,
feature_selection=selection_spec,
)
# one call renders the linked/signed features table + the consolidated target-gap table and
# returns everything later phases need (features, direction, target ids, gaps)
DASHBOARD_BASE_URL = "https://www.neuronpedia.org"
steering_base_url = DASHBOARD_BASE_URL
steering = display_steering_results(
pipeline_results,
tokenizer,
CONCEPT_TARGET_TOKENS,
neuronpedia_model=NEURONPEDIA_MODEL_ID,
neuronpedia_set=NEURONPEDIA_SOURCE_SET,
neuronpedia_base_url=steering_base_url,
min_layer=FEATURE_SELECTION_MIN_LAYER,
)
steered_features = steering.steered_features
pipeline_direction = steering.direction
target_a_id, target_b_id = steering.target_ids
fm_pre_gap, fm_post_gap = steering.pre_gap, steering.post_gap
assert fm_post_gap > fm_pre_gap, "feature-mediated steering should push the gap toward the target concept"
Phase 0: Precomputing activations and vectors
Precomputation completed in 1.83s
Found 18348 active features
Phase 1: Running forward pass
Forward pass completed in 0.57s
Phase 2: Building input vectors
Using 1 custom attribution targets with total weight 0.0000
Will include 8192 of 18348 feature nodes
Input vectors built in 0.82s
Phase 3: Computing logit attributions
1 logit attribution(s) completed in 0.22s
Phase 4: Computing feature attributions
Feature influence computation: 0%| | 0/8192 [00:00<?, ?it/s]
Feature influence computation: 3%|▎ | 256/8192 [00:00<00:06, 1254.64it/s]
Feature influence computation: 6%|▋ | 512/8192 [00:00<00:04, 1576.42it/s]
Feature influence computation: 9%|▉ | 768/8192 [00:00<00:04, 1749.84it/s]
Feature influence computation: 12%|█▎ | 1024/8192 [00:00<00:03, 1848.46it/s]
Feature influence computation: 16%|█▌ | 1280/8192 [00:00<00:03, 1958.33it/s]
Feature influence computation: 19%|█▉ | 1536/8192 [00:00<00:03, 2022.41it/s]
Feature influence computation: 22%|██▏ | 1792/8192 [00:00<00:03, 2031.55it/s]
Feature influence computation: 25%|██▌ | 2048/8192 [00:01<00:02, 2065.96it/s]
Feature influence computation: 28%|██▊ | 2304/8192 [00:01<00:02, 2077.82it/s]
Feature influence computation: 31%|███▏ | 2560/8192 [00:01<00:02, 2073.09it/s]
Feature influence computation: 34%|███▍ | 2816/8192 [00:01<00:02, 2069.36it/s]
Feature influence computation: 38%|███▊ | 3072/8192 [00:01<00:02, 2073.37it/s]
Feature influence computation: 41%|████ | 3328/8192 [00:01<00:02, 2147.68it/s]
Feature influence computation: 44%|████▍ | 3584/8192 [00:01<00:02, 2121.53it/s]
Feature influence computation: 47%|████▋ | 3840/8192 [00:01<00:02, 2104.37it/s]
Feature influence computation: 50%|█████ | 4096/8192 [00:02<00:01, 2070.49it/s]
Feature influence computation: 53%|█████▎ | 4352/8192 [00:02<00:02, 1891.75it/s]
Feature influence computation: 56%|█████▋ | 4608/8192 [00:02<00:01, 1936.97it/s]
Feature influence computation: 59%|█████▉ | 4864/8192 [00:02<00:01, 1925.88it/s]
Feature influence computation: 62%|██████▎ | 5120/8192 [00:02<00:01, 1966.50it/s]
Feature influence computation: 66%|██████▌ | 5376/8192 [00:02<00:01, 1794.42it/s]
Feature influence computation: 69%|██████▉ | 5632/8192 [00:02<00:01, 1892.18it/s]
Feature influence computation: 72%|███████▏ | 5888/8192 [00:03<00:01, 1954.71it/s]
Feature influence computation: 75%|███████▌ | 6144/8192 [00:03<00:01, 1990.22it/s]
Feature influence computation: 78%|███████▊ | 6400/8192 [00:03<00:01, 1723.94it/s]
Feature influence computation: 81%|████████▏ | 6656/8192 [00:03<00:00, 1812.45it/s]
Feature influence computation: 84%|████████▍ | 6912/8192 [00:03<00:00, 1881.53it/s]
Feature influence computation: 88%|████████▊ | 7168/8192 [00:03<00:00, 1920.00it/s]
Feature influence computation: 91%|█████████ | 7424/8192 [00:03<00:00, 1661.92it/s]
Feature influence computation: 94%|█████████▍| 7680/8192 [00:04<00:00, 1726.38it/s]
Feature influence computation: 97%|█████████▋| 7936/8192 [00:04<00:00, 1822.77it/s]
Feature influence computation: 100%|██████████| 8192/8192 [00:04<00:00, 1900.45it/s]
Feature influence computation: 100%|██████████| 8192/8192 [00:04<00:00, 1913.67it/s]
Feature attributions completed in 4.30s
Attribution completed in 10.14s
| Token | Pre prob | Post prob | Pre logit | Post logit | Δ |
|---|---|---|---|---|---|
| Fruit | 2.317% | 1.868% | 25.1250 | 29.3750 | +4.2500 |
| Color | 15.106% | 1.03e-05 | 27.0000 | 21.8750 | -5.1250 |
| Gap (Fruit − Color) | -1.8750 | +7.5000 | +9.3750 |
3. Feature semantics: the top-features table#
The steered features render as a table with the (layer, pos, feature) node tuple linked to its
dashboard, the signed influence score (Sign / |Score| columns — the signed_influence
selection can steer with negative-signed features, so the sign is colour-coded), and a best-effort
Explanation column resolved from the public Neuronpedia feature API.
# @title 3: Top-features table { display-mode: "form" }
# top_feature_ids are (layer, position, feature) tuples; dashboards/explanations are per
# (layer, feature), so collapse positions while preserving selection order
steered_layer_feature_pairs = list(dict.fromkeys((f[0], f[-1]) for f in steered_features))
# Best-effort explanation text from the public neuronpedia.org feature API; unmapped features
# simply render an empty Explanation cell
feature_explanations = resolve_feature_explanations(
model_id=NEURONPEDIA_MODEL_ID,
source_set=NEURONPEDIA_SOURCE_SET,
feature_tuples=steered_layer_feature_pairs,
base_url=DASHBOARD_BASE_URL,
)
display_top_features_comparison(
{"Steered Features (signed influence)": steered_features},
{"Steered Features (signed influence)": pipeline_results.top_feature_scores.tolist()},
neuronpedia_model=NEURONPEDIA_MODEL_ID,
neuronpedia_set=NEURONPEDIA_SOURCE_SET,
neuronpedia_base_url=DASHBOARD_BASE_URL,
show_score_sign=True,
feature_explanations=feature_explanations,
)
| # | Node | Sign | |Score| | Explanation |
|---|---|---|---|---|
| 1 | (25, 19, 16131) | − | 2.11e-08 | the comparison of two varieties of fruit, relating to size, taste, color, and genetic information |
| 2 | (24, 19, 13277) | − | 1.97e-08 | words related to questions and requests |
| 3 | (24, 19, 5999) | + | 1.08e-08 | language related to institutions, negative situations, the internet, and programming languages |
| 4 | (24, 19, 3865) | + | 5.31e-09 | dollar signs and other currency symbols, potentially alongside numbers or related terms like "terms" and "bonus". |
| 5 | (25, 19, 13210) | + | 5.17e-09 | grammatical structures and parts of speech like noun phrases and verb phrases |
4. Direct-hook path: concept direction -> hook-tensor steering#
Recomputes the embed-basis concept direction for the same concept pair (a consistency check against
the step 2 pipeline’s direction — cosine should be ~1.0 since both derive from the same
embedding-basis paired_rejection) and applies it.model_fwd_intervention(...) at
EMBED_INTERVENTION_HOOK in EMBED_INTERVENTION_MODE mode, comparing
pre/post_intervention_logits and the target-token gap against the feature-mediated result. The
same direction steered through selected transcoder features vs added directly at the hook point
produces different effect sizes — the feature-mediated path is typically stronger per unit scale.
# @title 4: Direct-hook steering { display-mode: "form" }
# Recompute the embed-basis concept direction for the same concept pair (no store rows -> embed basis)
direct_result = it.concept_direction(
module,
AnalysisBatch(
concept_group_a=fruits,
concept_group_b=colors,
concept_label="Concept: Fruit - Color (direct)",
concept_direction_mode="paired_rejection",
),
None,
0,
)
direct_direction = direct_result.concept_direction.detach()
cosine = torch.nn.functional.cosine_similarity(
pipeline_direction, direct_direction.float().cpu().reshape(-1), dim=0
).item()
print(f"pipeline-vs-direct direction cosine: {cosine:+.4f} (~1.0 expected — same embed-basis construction)")
# Direct hook-tensor intervention at the canonical hook point
module.analysis_cfg = AnalysisCfg(target_op=it.model_fwd_intervention, ignore_manual=True, save_tokens=False)
init_analysis_cfgs(module, [module.analysis_cfg])
# chat-rendered prompts already carry their special tokens; plain completion prompts need them added
enc = tokenizer(prompt, return_tensors="pt", padding=False, add_special_tokens=not CHAT_FORMAT_PROMPT)
device = next(module.model.parameters()).device
if BACKEND == "transformerlens":
# HookedTransformer.forward takes `input`, not the HF-style `input_ids`/`attention_mask` keys
batch = {"input": enc["input_ids"].to(device)}
else:
batch = {k: (v.to(device) if isinstance(v, torch.Tensor) else v) for k, v in dict(enc).items()}
# legacy HookedTransformer models (the CT TransformerLens backend) expose no `unembed.hook_in`;
# their pre-unembed equivalent is `ln_final.hook_normalized` (alias-map expansion tracked in
# interpretune#223)
intervention_hook = EMBED_INTERVENTION_HOOK
if BACKEND == "transformerlens" and EMBED_INTERVENTION_HOOK == "unembed.hook_in":
intervention_hook = "ln_final.hook_normalized"
direct_batch = AnalysisBatch(
prompts=[prompt],
concept_direction=direct_direction,
logit_target_ids=torch.tensor([target_a_id], dtype=torch.long),
concept_group_a_token_ids=[target_a_id],
concept_group_b_token_ids=[target_b_id],
concept_cache_key=intervention_hook,
intervention_hook_pattern=intervention_hook,
intervention_mode=EMBED_INTERVENTION_MODE,
direction_scale_factor=INTERVENTION_SCALE_FACTOR,
)
direct_out = it.model_fwd_intervention(module, direct_batch, batch, 0)
direct_pre_gap, direct_post_gap = display_target_gap(
direct_out.pre_intervention_logits.float().cpu().reshape(-1),
direct_out.post_intervention_logits.float().cpu().reshape(-1),
(CONCEPT_TARGET_TOKENS[0], target_a_id),
(CONCEPT_TARGET_TOKENS[1], target_b_id),
title="Direct-hook steering — target gap",
)
print(
f"feature-mediated delta {fm_post_gap - fm_pre_gap:+.3f} vs direct-hook delta "
f"{direct_post_gap - direct_pre_gap:+.3f}"
)
assert direct_post_gap > direct_pre_gap, "direct-hook steering should push the gap toward the target concept"
pipeline-vs-direct direction cosine: +1.0000 (~1.0 expected — same embed-basis construction)
feature-mediated delta +9.375 vs direct-hook delta +2.000
| Token | Pre prob | Post prob | Pre logit | Post logit | Δ |
|---|---|---|---|---|---|
| Fruit | 2.317% | 2.036% | 25.1250 | 27.8750 | +2.7500 |
| Color | 15.106% | 1.797% | 27.0000 | 27.7500 | +0.7500 |
| Gap (Fruit − Color) | -1.8750 | +0.1250 | +2.0000 |
5. Input/output decoupling analysis (graph hydration + UMAP)#
Attribution-selected steering features are chosen for their output effect (decoder projection
onto the target logit difference), while dashboard explanations describe their input behavior
(the contexts they fire on) — the two can decouple sharply. A recurring special case is the
suppressor-motif exemplar: a feature that fires on concept contexts yet projects against the
concept token (redundancy suppression under next-token training) — causally ideal for sign-aware
steering, semantically confusing on its dashboard. This step demonstrates those mechanics
directly, and in doing so demos the framework’s graph hydration capability
(analysis_backend.hydrate_graph_from_batch) for detailed post-hoc analysis of a persisted
attribution result:
hydrate the step-2 attribution graph; highlight the prompt’s concept-token positions;
compute each analyzed feature’s input profile (activation mass at concept positions) and output profile (signed decoder projection onto the unit target-token unembed difference) via the shared
feature_io_profileshelper;render the decoupling table (signature column flags
decoupledandsuppressor-motifrows);project decoder vectors to 2D (UMAP, PCA fallback — tooling shared with the latent-dynamics notebooks) with hover details per analyzed feature. Axis tick numbers are intentionally hidden: UMAP coordinates are non-metric (arbitrary rotation/scale; only local neighborhood structure is meaningful).
Expect roughly 1 of the 5 attribution picks to be input-aligned — a fruit-context feature that is
also a suppressor-motif exemplar (historically L25/16131, with all-Fruit negative logits) — and
the rest to be decoupled output machinery.
# @title 5: Decoupling analysis via hydrated graph + UMAP { display-mode: "form" }
import numpy as np
from interpretune.analysis.backends import require_analysis_backend
from it_examples.utils.example_helpers import concept_token_positions, feature_io_profiles
from it_examples.utils.nb_ui_utils import (
display_concept_positions,
display_feature_decoupling_table,
plot_decoder_projection_map,
)
analysis_backend = require_analysis_backend(module)
graph = analysis_backend.hydrate_graph_from_batch(pipeline_results)
# concept-token positions derived from the demo's own concept groups + probe/target tokens
concept_words = {w.lower() for w in (*fruits, *colors, *CONCEPT_TARGET_TOKENS, "orange")}
prompt_token_ids = [int(t) for t in graph.input_tokens]
concept_positions = concept_token_positions(tokenizer, prompt_token_ids, sorted(concept_words))
display_concept_positions(tokenizer, prompt_token_ids, concept_positions)
embed_weight = analysis_backend.get_embedding_weight(module).detach().float().cpu()
target_diff = embed_weight[target_a_id] - embed_weight[target_b_id]
target_diff = target_diff / target_diff.norm()
target_label = f"{CONCEPT_TARGET_TOKENS[0]}-{CONCEPT_TARGET_TOKENS[1]}"
analyzed_pairs = list(dict.fromkeys((int(f[0]), int(f[-1])) for f in steered_features))
all_explanations = dict(feature_explanations)
transcoder_set = getattr(module.replacement_model.transcoders, "_module", module.replacement_model.transcoders)
profiles = feature_io_profiles(graph, analyzed_pairs, target_diff, transcoder_set, concept_positions)
display_feature_decoupling_table(profiles, all_explanations, target_label=target_label)
# decoder vectors for the interactive projection map (analyzed + random active-feature background)
rng = np.random.default_rng(17)
active_rows = graph.active_features.cpu()
background_pool = sorted({(int(r[0]), int(r[2])) for r in active_rows} - set(analyzed_pairs))
background_idx = rng.choice(len(background_pool), size=min(300, len(background_pool)), replace=False)
background_pairs = [background_pool[i] for i in background_idx]
def _decoder_rows(pairs):
return torch.stack(
[transcoder_set._get_decoder_vectors(lyr, torch.tensor([ft]))[0].detach().float().cpu() for lyr, ft in pairs]
)
plot_decoder_projection_map(
profiles,
_decoder_rows(analyzed_pairs),
_decoder_rows(background_pairs),
feature_explanations=all_explanations,
target_label=target_label,
title="Steering-feature decoder map",
)
| Feature | Input concept share | Act mass | Output proj (Fruit-Color) | Signature | Explanation |
|---|---|---|---|---|---|
| L25/16131 | 0.481 | 346.72 | -0.2961 | suppressor-motif | the comparison of two varieties of fruit, relating to size, taste, color, and ge |
| L24/3865 | 0.097 | 191.47 | +0.0264 | dollar signs and other currency symbols, potentially alongside numbers or relate | |
| L25/13210 | 0.000 | 33.75 | +0.0227 | grammatical structures and parts of speech like noun phrases and verb phrases | |
| L24/5999 | 0.347 | 923.50 | +0.0121 | language related to institutions, negative situations, the internet, and program | |
| L24/13277 | 0.186 | 1297.00 | +0.0061 | words related to questions and requests |
/mnt/cache/speediedan/.venvs/it_latest/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
warn(
Summary#
Feature-mediated and direct-hook steering paths on one proven example, one backend per run — both driven by the same embed-basis concept direction (store-basis directions are a
tests/nb_experimentsresearch thread, seeEXPERIMENT_STATUS.md).Semantic grounding via public neuronpedia.org feature dashboards and explanations.
Input/output decoupling mechanics demonstrated via graph hydration + decoder-space UMAP, doubling as a demo of persisted-graph post-hoc analysis.
Local Neuronpedia dashboards, locally generated explanations, and user-curated feature steering:
ct_concept_steering_demo_local_np.ipynb.Deeper coverage of the underlying op pipeline (per-op invocation, native + hub composition): see
ct_analysis_backend_demo.ipynb. Capability map:tests/nb_experiments/intervention_capabilities_overview.md.