Interpretune SAELens Tutorial#
{height=”55px” width=”401px”}
Intro#
Interpretune is a flexible framework for exploring, analyzing and tuning llm world models. In this tutorial, we’ll walk through a simple example of using Interpretune to pursue interpretability research with SAELens. As we’ll see, Interpretune handles the required execution context composition, allowing us to use the same code in a variety of contexts, depending upon the level of abstraction required.
As a long-time PyTorch and PyTorch Lightning contributor, I’ve found the PyTorch Lightning framework is the right level
of abstraction for a large variety of ML research contexts, but some contexts benefit from using core PyTorch directly.
Additionally, some users may prefer to use the core PyTorch framework directly for a wide variety of reasons including
maximizing portability. As will be demonstrated here, Interpretune maximizes flexibility and portability by adhering to
a well-defined protocol that allows auto-composition of our research module with the adapters required for execution in
a wide variety of contexts. In this example, we’ll be executing the same module with core PyTorch and PyTorch Lightning,
demonstrating the use of SAELens w/ Interpretune for interpretability research.
Note - this is a WIP, but this is the core idea. If you have any feedback, please let me know!
A note on memory usage#
In these exercises, we’ll be loading some pretty large opls into memory (e.g. Gemma 2-2B and its SAEs, as well as a host of other models in later sections of the material). It’s useful to have functions which can help profile memory usage for you, so that if you encounter OOM errors you can try and clear out unnecessary models. For example, we’ve found that with the right memory handling (i.e. deleting models and objects when you’re not using them any more) it should be possible to run all the exercises in this material on a Colab Pro notebook, and all the exercises minus the handful involving Gemma on a free Colab notebook.
See this dropdown for some functions which you might find helpful, and how to use them.
First, we can run some code to inspect our current memory usage. Here’s me running this code during the exercise set on SAE circuits, after having already loaded in the Gemma models from the previous section. This was on a Colab Pro notebook.
# Profile memory usage, and delete gemma models if we've loaded them in
namespace = globals().copy() | locals()
part32_utils.profile_pytorch_memory(namespace=namespace, filter_device="cuda:0")
Allocated = 35.88 GB Total = 39.56 GB Free = 3.68 GB ┌──────────────────────┬────────────────────────┬──────────┬─────────────┐ │ Name │ Object │ Device │ Size (GB) │ ├──────────────────────┼────────────────────────┼──────────┼─────────────┤ │ gemma_2_2b │ HookedSAETransformer │ cuda:0 │ 11.94 │ │ gpt2 │ HookedSAETransformer │ cuda:0 │ 0.61 │ │ gemma_2_2b_sae │ SAE │ cuda:0 │ 0.28 │ │ sae_resid_dirs │ Tensor (4, 24576, 768) │ cuda:0 │ 0.28 │ │ gpt2_sae │ SAE │ cuda:0 │ 0.14 │ │ logits │ Tensor (4, 15, 50257) │ cuda:0 │ 0.01 │ │ logits_with_ablation │ Tensor (4, 15, 50257) │ cuda:0 │ 0.01 │ │ clean_logits │ Tensor (4, 15, 50257) │ cuda:0 │ 0.01 │ │ _ │ Tensor (16, 128, 768) │ cuda:0 │ 0.01 │ │ clean_sae_acts_post │ Tensor (4, 15, 24576) │ cuda:0 │ 0.01 │ └──────────────────────┴────────────────────────┴──────────┴─────────────┘
From this, we see that we’ve allocated a lot of memory for the the Gemma model, so let’s delete it. We’ll also run some code to move any remaining objects on the GPU which are larger than 100MB to the CPU, and print the memory status again.
del gemma_2_2b
del gemma_2_2b_sae
THRESHOLD = 0.1 # GB
for obj in gc.get_objects():
try:
if isinstance(obj, torch.nn.Module) and part32_utils.get_tensors_size(obj) / 1024**3 > THRESHOLD:
if hasattr(obj, "cuda"):
obj.cpu()
if hasattr(obj, "reset"):
obj.reset()
except:
pass
# Move our gpt2 model & SAEs back to GPU (we'll need them for the exercises we're about to do)
gpt2.to(device)
gpt2_saes = {layer: sae.to(device) for layer, sae in gpt2_saes.items()}
part32_utils.print_memory_status()
Allocated = 14.90 GB Reserved = 39.56 GB Free = 24.66
Mission success! We’ve managed to free up a lot of memory. Note that the code which moves all objects collected by the garbage collector to the CPU is often necessary to free up the memory. We can’t just delete the objects directly because PyTorch can still sometimes keep references to them (i.e. their tensors) in memory. In fact, if you add code to the for loop above to print out obj.shape when obj is a tensor, you’ll see that a lot of those tensors are actually Gemma model weights, even once you’ve deleted gemma_2_2b.
# Parameters - These will be injected by papermill during parameterized test runs
core_log_dir = None # Directory to save analysis logs (if None, a temp directory will be created)
Imports#
# Core imports
import interpretune as it # registered analysis ops will be available as it.<op> when analysis is imported
from transformer_lens import ActivationCache # noqa: F401
from tabulate import tabulate
from it_examples import _ACTIVE_PATCHES # noqa: F401 # TODO: add note about this unless patched in SL before release
from it_examples.example_module_registry import MODULE_EXAMPLE_REGISTRY # TODO: move to hub once implemented
from interpretune import ITSessionConfig, ITSession, SAELensFromPretrainedConfig, LatentAnalysisTargets
Configure our IT Session#
Here we define or customize our session configuration, which includes:
Experiment/task module and datamodule (in this case, ‘rte’ for the RTE task)
We can customize any module, datamodule, or adapter-specific configuration options we want to use. In this case, we set target
sae_cfgsthat we want to use for our analysis. We also could customize generation parameters, tokenization, the pretrained/config-based model we want to use (in this case, GPT2) etc.
The adapter context we want to use. In this case,
corePyTorch (vs e.g. Lightning) andsae_lens(vs e.g.transformer_lens).
When an ITSession is created, the selected adapter context will trigger composition of the relevant adapters with our experiment/task module and datamodule. The intention of this abstraction is to enable the same experiment/task logic to be used unchanged across a broad variety of PyTorch framework and analytical package contexts.
# Load our demo config (this will be done from the hub once that is available)
base_itdm_cfg, base_it_cfg, dm_cls, m_cls = MODULE_EXAMPLE_REGISTRY.get("gpt2.rte_demo.sae_lens")
# Optionally override base_it_cfg.core_log_dir with the notebook parameter if provided
if core_log_dir:
base_it_cfg.core_log_dir = core_log_dir
# update our config with our desired SAE analysis targets
sae_targets = LatentAnalysisTargets(sae_release="gpt2-small-hook-z-kk", target_layers=[9, 10])
sae_cfgs = [
SAELensFromPretrainedConfig(release=sae_fqn.release, sae_id=sae_fqn.sae_id)
for sae_fqn in sae_targets.latent_model_fqns
]
base_it_cfg.sae_cfgs = sae_cfgs
# configure our session with our desired adapter composition, core and sae_lens in this case
session_cfg = ITSessionConfig(
adapter_ctx=(it.Adapter.core, it.Adapter.sae_lens),
datamodule_cfg=base_itdm_cfg,
module_cfg=base_it_cfg,
datamodule_cls=dm_cls,
module_cls=m_cls,
)
# start our session
it_session = ITSession(session_cfg)
Moving model to device: cuda
[INFO] interpretune.utils.logging: Moving SAETransformerBridge to device: cuda
INFO:interpretune.utils.logging:Moving SAETransformerBridge to device: cuda
[INFO] interpretune.utils.logging: Attempted to clean a key that was not present, continuing without cleaning that key: 'GPT2Config' object has no attribute 'quantization_config'
INFO:interpretune.utils.logging:Attempted to clean a key that was not present, continuing without cleaning that key: 'GPT2Config' object has no attribute 'quantization_config'
[INFO] interpretune.utils.logging: Attempted to clean a key that was not present, continuing without cleaning that key: 'GPT2Config' object has no attribute '_pre_quantization_dtype'
INFO:interpretune.utils.logging:Attempted to clean a key that was not present, continuing without cleaning that key: 'GPT2Config' object has no attribute '_pre_quantization_dtype'
Run Demo Analysis#
Define Our Analysis Run#
We define what analysis we want to run. This includes defining:
our latent space targets (latent_analysis_targets in this case)
one or more analysis configurations (which can use manual or generated analysis steps)
the analysis runner
The AnalysisRunner is the core component of Interpretune that handles the execution of our analysis. It takes care of running the analysis operations defined in our analysis set, managing the execution context, and storing the results.
from interpretune import AnalysisRunner, AnalysisCfg, AnalysisStore
# Define our `AnalysisRunner`. We set:
# 1. our analysis targets across all analysis configurations we want to run in the next analysis run
# 2. batch and epoch limits
# 3. ignore any manual `analysis_step` in our provided module because we want to generate analysis steps based on
# provided operations
run_kwargs = dict(latent_analysis_targets=sae_targets, it_session=it_session, max_epochs=1)
run_config = dict(limit_analysis_batches=3, ignore_manual=True, **run_kwargs)
runner = AnalysisRunner(run_cfg=run_config)
# Define our Analysis Configurations
# here we demo a few different op compositions involving logit differences
auto_logit_diffs_base_cfg = AnalysisCfg(target_op=it.logit_diffs_base, save_prompts=False, save_tokens=False)
auto_logit_diffs_sae_cfg = AnalysisCfg(target_op=it.logit_diffs_sae, save_prompts=True, save_tokens=True)
auto_logit_diffs_attr_grad_cfg = AnalysisCfg(target_op=it.logit_diffs_attr_grad, save_prompts=True, save_tokens=True)
auto_logit_diffs_attr_ablation_cfg = AnalysisCfg(
target_op=it.logit_diffs_attr_ablation, save_prompts=False, save_tokens=False
)
[INFO] interpretune.utils.logging: Preparing data: InterpretunableDataModule
INFO:interpretune.utils.logging:Preparing data: InterpretunableDataModule
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
INFO:interpretune.utils.logging:The following columns don't have a corresponding argument in `SAETransformerBridge.forward` and have been ignored: hypothesis, idx, label, premise, sequences. If hypothesis, idx, label, premise, sequences are not expected by `SAETransformerBridge.forward`, you can safely ignore this message.
[INFO] interpretune.utils.logging: Setting up datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Setting up datamodule: InterpretunableDataModule
[INFO] interpretune.utils.logging: Setting up model: InterpretunableModule
INFO:interpretune.utils.logging:Setting up model: InterpretunableModule
[INFO] interpretune.utils.logging: initializing optimizers and schedulers: InterpretunableModule
INFO:interpretune.utils.logging:initializing optimizers and schedulers: InterpretunableModule
[INFO] interpretune.utils.logging: Setting input require grads not currently supported by BaseSAELensModule.
INFO:interpretune.utils.logging:Setting input require grads not currently supported by BaseSAELensModule.
Run the Analysis#
analysis_results = runner.run_analysis(
analysis_cfgs=(
auto_logit_diffs_base_cfg,
auto_logit_diffs_sae_cfg,
auto_logit_diffs_attr_grad_cfg,
auto_logit_diffs_attr_ablation_cfg,
)
)
/home/speediedan/repos/interpretune/src/interpretune/base/components/mixins.py:80: Analysis configuration has not been set.
[INFO] interpretune.utils.logging: Running analysis start hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis start hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis epoch end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis epoch end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT module: InterpretunableModule
INFO:interpretune.utils.logging:Running stage end hooks on IT module: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Running stage end hooks on IT datamodule: InterpretunableDataModule
[INFO] interpretune.utils.logging: Running analysis start hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis start hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis epoch end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis epoch end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT module: InterpretunableModule
INFO:interpretune.utils.logging:Running stage end hooks on IT module: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Running stage end hooks on IT datamodule: InterpretunableDataModule
[INFO] interpretune.utils.logging: Running analysis start hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis start hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis epoch end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis epoch end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT module: InterpretunableModule
INFO:interpretune.utils.logging:Running stage end hooks on IT module: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Running stage end hooks on IT datamodule: InterpretunableDataModule
[INFO] interpretune.utils.logging: Running analysis start hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis start hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis epoch end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis epoch end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running analysis end hooks: InterpretunableModule
INFO:interpretune.utils.logging:Running analysis end hooks: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT module: InterpretunableModule
INFO:interpretune.utils.logging:Running stage end hooks on IT module: InterpretunableModule
[INFO] interpretune.utils.logging: Running stage end hooks on IT datamodule: InterpretunableDataModule
INFO:interpretune.utils.logging:Running stage end hooks on IT datamodule: InterpretunableDataModule
Set convenience variables for exploratory analysis#
run_cfg = runner.run_cfg
sl_test_module = run_cfg.module # convenience handle to the module used in the analysis
artifact_cfg = run_cfg.artifact_cfg
# Set tutorial_active_ops based on the keys in analysis_results
if isinstance(analysis_results, dict):
tutorial_active_ops = set(analysis_results.keys())
elif hasattr(run_cfg, "analysis_cfg") and run_cfg.analysis_cfg:
# Single analysis configuration
tutorial_active_ops = {run_cfg.analysis_cfg.name}
else:
tutorial_active_ops = set()
# If analysis_results is an AnalysisStore, convert to a dict with a single entry
if isinstance(analysis_results, AnalysisStore):
analysis_results = {run_cfg.analysis_cfg.name: analysis_results}
print(f"Analysis completed for {len(analysis_results) if isinstance(analysis_results, dict) else 1} operations:")
for cfg_name in analysis_results.keys() if isinstance(analysis_results, dict) else [run_cfg.analysis_cfg.name]:
print(f"- {cfg_name}")
Analysis completed for 4 operations:
- logit_diffs_base
- logit_diffs_sae
- logit_diffs_attr_grad
- logit_diffs_attr_ablation
Review Demo Results#
Clean vs SAE Sample-wise Logit Diffs#
if {it.logit_diffs_base.name, it.logit_diffs_sae.name}.issubset(tutorial_active_ops):
from interpretune.analysis import base_vs_sae_logit_diffs
base_vs_sae_logit_diffs(
sae=analysis_results[it.logit_diffs_sae.name],
base_ref=analysis_results[it.logit_diffs_base.name],
top_k=artifact_cfg.top_k_clean_logit_diffs,
tokenizer=sl_test_module.datamodule.tokenizer,
)
+-------------+----------------------------------------------------------------------------------+----------+--------------------+------------------+
| Sample ID | Prompt | Answer | Clean Logit Diff | SAE Logit Diff |
+=============+==================================================================================+==========+====================+==================+
| 0 | Dana Reeve, the widow of the actor Christopher Reeve, has died of lung cancer at | No | +1.081 | +1.292 |
| | age 44, according to the Christopher Reeve Foundation.Does the previous passage | | | |
| | imply that Christopher Reeve had an accident.? Answer with only one word, either | | | |
| | Yes or No. | | | |
+-------------+----------------------------------------------------------------------------------+----------+--------------------+------------------+
| 3 | The Amish community in Pennsylvania, which numbers about 55,000, lives an | No | +0.717 | +0.669 |
| | agrarian lifestyle, shunning technological advances like electricity and | | | |
| | automobiles. And many say their insular lifestyle gives them a sense that they | | | |
| | are protected from the violence of American society. But as residents gathered | | | |
| | near the school, some wearing traditional garb and arriving in horse-drawn | | | |
| | buggies, they said that sense of safety had been shattered. "If someone snaps | | | |
| | and wants to do something stupid, there's no distance that's going to stop | | | |
| | them," said Jake King, 56, an Amish lantern maker who knew several families | | | |
| | whose children had been shot.Does the previous passage imply that Pennsylvania | | | |
| | has the biggest Amish community in the U.S.? Answer with only one word, either | | | |
| | Yes or No. | | | |
+-------------+----------------------------------------------------------------------------------+----------+--------------------+------------------+
| 1 | Yet, we now are discovering that antibiotics are losing their effectiveness | Yes | +0.324 | +0.092 |
| | against illness. Disease-causing bacteria are mutating faster than we can come | | | |
| | up with new antibiotics to fight the new variations.Does the previous passage | | | |
| | imply that Bacteria is winning the war against antibiotics.? Answer with only | | | |
| | one word, either Yes or No. | | | |
+-------------+----------------------------------------------------------------------------------+----------+--------------------+------------------+
/home/speediedan/repos/interpretune/src/interpretune/analysis/core.py:532: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
Proportion Correct Answers on Dataset By Analysis Op#
from interpretune.analysis import compute_correct
pred_summaries = {op: compute_correct(summ, op) for op, summ in analysis_results.items()}
table_rows = []
for op, (total_correct, percentage_correct, _) in pred_summaries.items():
table_rows.append([op, total_correct, f"{percentage_correct:.2f}%"])
print(tabulate(table_rows, headers=["Op", "Total Correct", "Percentage Correct"], tablefmt="grid"))
+---------------------------+-----------------+----------------------+
| Op | Total Correct | Percentage Correct |
+===========================+=================+======================+
| logit_diffs_base | 3 | 50.00% |
+---------------------------+-----------------+----------------------+
| logit_diffs_sae | 4 | 66.67% |
+---------------------------+-----------------+----------------------+
| logit_diffs_attr_grad | 4 | 66.67% |
+---------------------------+-----------------+----------------------+
| logit_diffs_attr_ablation | 4 | 66.67% |
+---------------------------+-----------------+----------------------+
/home/speediedan/repos/interpretune/src/interpretune/analysis/core.py:532: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
/mnt/cache/speediedan/.venvs/it_latest/lib/python3.13/site-packages/datasets/formatting/torch_formatter.py:93: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
return torch.tensor(value, **{**default_dtype, **self.torch_tensor_kwargs})
Per Batch Ablation Effect Graphs [Optional]#
if artifact_cfg.latent_effects_graphs and it.logit_diffs_attr_ablation.name in tutorial_active_ops:
# TODO: add note that only latent effects associated with correct answers currently displayed
# TODO: allow toggling correct filtering during runs
analysis_results[it.logit_diffs_attr_ablation.name].plot_latent_effects(
per_batch=artifact_cfg.latent_effects_graphs_per_batch
)
Per-SAE Ablation Effects#
if it.logit_diffs_attr_ablation.name in tutorial_active_ops:
ablation_batch_preds = pred_summaries[it.logit_diffs_attr_ablation.name].batch_predictions
activation_summary = analysis_results[it.logit_diffs_sae.name].calc_activation_summary()
ablation_metrics = analysis_results[it.logit_diffs_attr_ablation.name].calculate_latent_metrics(
pred_summ=pred_summaries[it.logit_diffs_attr_ablation.name],
activation_summary=activation_summary,
# filter_by_correct=True,
run_name="logit_diffs.attribution.ablation",
)
tables = ablation_metrics.create_attribution_tables(
top_k=artifact_cfg.top_k_latents_table, filter_type="both", per_latent_model=artifact_cfg.table_per_latent_model
)
for title, table in tables.items():
print(f"\n{title}\n{table}\n")
sl_test_module.display_latent_dashboards(
ablation_metrics,
title="Ablation-Mediated Latent Analysis",
sae_release=runner.run_cfg.latent_analysis_targets.sae_release,
top_k=artifact_cfg.top_k_latent_dashboards,
)
Top 2 positive total_effect for blocks.9.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.9.attn.hook_z.hook_sae_acts_post | 9464 | 0.1497 | 0.0374 | 1 | 1.3833 | 4 |
| blocks.9.attn.hook_z.hook_sae_acts_post | 1695 | 0.056 | 0.014 | 0.75 | 0.3871 | 3 |
Top 2 negative total_effect for blocks.9.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.9.attn.hook_z.hook_sae_acts_post | 15185 | -0.0212 | -0.0053 | 0.75 | 0.3153 | 3 |
| blocks.9.attn.hook_z.hook_sae_acts_post | 7090 | -0.0182 | -0.0045 | 0.5 | 0.1243 | 2 |
Top 2 positive total_effect for blocks.10.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:-----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.10.attn.hook_z.hook_sae_acts_post | 21019 | 0.2374 | 0.0594 | 1 | 4.3174 | 4 |
| blocks.10.attn.hook_z.hook_sae_acts_post | 22076 | 0.1583 | 0.0396 | 1 | 1.5017 | 4 |
Top 2 negative total_effect for blocks.10.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:-----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.10.attn.hook_z.hook_sae_acts_post | 8637 | -0.1551 | -0.0388 | 1 | 0.3525 | 4 |
| blocks.10.attn.hook_z.hook_sae_acts_post | 20608 | -0.119 | -0.0297 | 0.75 | 0.6117 | 3 |
Ablation-Mediated Latent Analysis for blocks.9.attn.hook_z.hook_sae_acts_post:
positive:
#9464 had total effect 0.15 and was active in 4 examples
https://neuronpedia.org/gpt2-small/9-att-kk/9464?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
negative:
#15185 had total effect -0.02 and was active in 3 examples
https://neuronpedia.org/gpt2-small/9-att-kk/15185?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
Ablation-Mediated Latent Analysis for blocks.10.attn.hook_z.hook_sae_acts_post:
positive:
#21019 had total effect 0.24 and was active in 4 examples
https://neuronpedia.org/gpt2-small/10-att-kk/21019?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
negative:
#8637 had total effect -0.16 and was active in 4 examples
https://neuronpedia.org/gpt2-small/10-att-kk/8637?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
/home/speediedan/repos/interpretune/src/interpretune/analysis/core.py:532: UserWarning:
To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
Per-SAE Attribution Patching Effects#
if it.logit_diffs_attr_grad.name in tutorial_active_ops:
# per-latent-model activation summaries are calculated using our AnalysisStore since the relevant keys are present,
# no need to provide a separate activation summary from another comparison cache in this case as with ablation
activation_summary = analysis_results[it.logit_diffs_attr_grad.name].calc_activation_summary()
attribution_patching_metrics = analysis_results[it.logit_diffs_attr_grad.name].calculate_latent_metrics(
pred_summ=pred_summaries[it.logit_diffs_attr_grad.name], run_name="logit_diffs.attribution.grad_based"
)
tables = attribution_patching_metrics.create_attribution_tables(
top_k=artifact_cfg.top_k_latents_table, filter_type="both", per_latent_model=artifact_cfg.table_per_latent_model
)
for title, table in tables.items():
print(f"\n{title}\n{table}\n")
sl_test_module.display_latent_dashboards(
attribution_patching_metrics,
title="Attribution Patching-Mediated Latent Analysis",
sae_release=runner.run_cfg.latent_analysis_targets.sae_release,
top_k=artifact_cfg.top_k_latent_dashboards,
)
Top 2 positive total_effect for blocks.9.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.9.attn.hook_z.hook_sae_acts_post | 9464 | 0.1539 | 0.0385 | 1 | 1.3833 | 4 |
| blocks.9.attn.hook_z.hook_sae_acts_post | 1695 | 0.0576 | 0.0144 | 0.75 | 0.3871 | 3 |
Top 2 negative total_effect for blocks.9.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.9.attn.hook_z.hook_sae_acts_post | 15185 | -0.0216 | -0.0054 | 0.75 | 0.3153 | 3 |
| blocks.9.attn.hook_z.hook_sae_acts_post | 7090 | -0.0184 | -0.0046 | 0.5 | 0.1243 | 2 |
Top 2 positive total_effect for blocks.10.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:-----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.10.attn.hook_z.hook_sae_acts_post | 21019 | 0.2488 | 0.0622 | 1 | 4.3174 | 4 |
| blocks.10.attn.hook_z.hook_sae_acts_post | 22076 | 0.1601 | 0.04 | 1 | 1.5017 | 4 |
Top 2 negative total_effect for blocks.10.attn.hook_z.hook_sae_acts_post
| Hook | Latent Index | Total Effect | Mean Effect | Proportion Active | Mean Activation | Number Active |
|:-----------------------------------------|---------------:|---------------:|--------------:|--------------------:|------------------:|----------------:|
| blocks.10.attn.hook_z.hook_sae_acts_post | 8637 | -0.1551 | -0.0388 | 1 | 0.3525 | 4 |
| blocks.10.attn.hook_z.hook_sae_acts_post | 20608 | -0.1176 | -0.0294 | 0.75 | 0.6117 | 3 |
Attribution Patching-Mediated Latent Analysis for blocks.9.attn.hook_z.hook_sae_acts_post:
positive:
#9464 had total effect 0.15 and was active in 4 examples
https://neuronpedia.org/gpt2-small/9-att-kk/9464?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
negative:
#15185 had total effect -0.02 and was active in 3 examples
https://neuronpedia.org/gpt2-small/9-att-kk/15185?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
Attribution Patching-Mediated Latent Analysis for blocks.10.attn.hook_z.hook_sae_acts_post:
positive:
#21019 had total effect 0.25 and was active in 4 examples
https://neuronpedia.org/gpt2-small/10-att-kk/21019?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
negative:
#8637 had total effect -0.16 and was active in 4 examples
https://neuronpedia.org/gpt2-small/10-att-kk/8637?embed=true&embedexplanation=true&embedplots=true&embedtest=true&height=300
Per-SAE Ablation vs Attribution-Patching Effect Parity#
if {it.logit_diffs_attr_grad.name, it.logit_diffs_attr_ablation.name}.issubset(tutorial_active_ops):
from interpretune.analysis import latent_metrics_scatter
# Visualize results for each hook
# Call the function with our metrics
latent_metrics_scatter(
ablation_metrics, attribution_patching_metrics, label1="Ablation", label2="Attribution Patching"
)
Cross-Backend Composition#
The AnalysisStore results produced above are fully backend-agnostic — they can be saved to disk and combined with results from other analysis backends. For a demonstration of cross-backend composition combining SAE-Lens analysis with circuit-tracer attribution, see the Concept-Direction Steering Demo.