interpretune.utils#
- exception interpretune.utils.ITInstantiationFeedbackWarning[source]#
Config-normalization feedback emitted while instantiating Interpretune configs.
These messages (e.g. TL
tokenizer_namefallbacks, auto-composition notices) are informative when a user instantiates a config directly, but bulk registry hydration (gen_module_registry) instantiates EVERY registered entry, emitting per-entry feedback irrelevant to the caller’s requested entry. Bulk-hydration paths therefore suppress exactly this category — never blanket-filter warnings — so direct instantiation keeps the feedback while registry access stays quiet.
- exception interpretune.utils.MisconfigurationException[source]#
Exception used to inform users of misuse with interpretune.
- interpretune.utils.NeuronpediaDBError#
alias of
NeuronpediaLocalDBImportError
- exception interpretune.utils.NeuronpediaExplanationError[source]#
Raised when explanation artifact generation fails.
- class interpretune.utils.LocalNeuronpediaServiceStatus(webapp_url, webapp_available, webapp_status_code, webapp_error, db_url_redacted, db_available, db_error)[source]#
Availability snapshot for the local Neuronpedia webapp and Postgres services.
- class interpretune.utils.NeuronpediaDashboardLayerResult(layer_num, output_dir, export_root, import_summary, elapsed_seconds, skipped=False)[source]#
Result metadata for one generated layer.
- class interpretune.utils.NeuronpediaDashboardPipelineConfig(*, model_name, model_layers, sae_set, neuronpedia_source_set_id, neuronpedia_source_set_description, creator_name, release_id, release_title, release_url, hf_weights_repo_id, hf_weights_path_template, hook_point, prompts_huggingface_dataset_path, start_layer, end_layer, layer_list=None, sae_path_template, prompts_dataset_mode='load_dataset', hf_model_path=None, prompts_huggingface_dataset_config_name=None, prompts_huggingface_dataset_split=None, prompts_dataset_text_field=None, prompts_pretokenized_dataset_path=None, prompts_shared_tokens_file=None, run_root=PosixPath('/home/docs/.cache/huggingface/interpretune/neuronpedia/dashboard_runs'), run_name_suffix=None, export_root=<factory>, existing_log_path=None, pipeline_log_path=None, worker_id=None, enable_layer_locks=False, layer_lock_stale_seconds=0, saedashboard_repo_root=<factory>, saelens_repo_root=<factory>, neuronpedia_utils_root=<factory>, interpretune_env_file=<factory>, python_executable='/home/docs/checkouts/readthedocs.org/user_builds/interpretune/envs/238/bin/python', use_skip_transcoder=False, sae_dtype='float32', model_dtype='bfloat16', sparsity_threshold=1, n_prompts_total=24576, n_tokens_in_prompt=128, n_features_per_batch=128, n_prompts_in_forward_pass=32, primary_acts_batch_size=None, start_batch=0, end_batch=None, zero_out_bos_token=False, use_clt=False, clt_dtype='', clt_weights_filename='', dataset_streaming=True, model_wrapper='hooked', bridge_enable_compatibility_mode=True, bridge_compatibility_mode_kwargs=<factory>, runner_log_resource_snapshots=False, runner_log_hook_aliases=False, runner_log_performance=False, runner_profile_rolling_substages=False, runner_shuffle_tokens=True, runner_implementation='current', runner_cleanup_each_minibatch=False, runner_correlation_accumulation_device='auto', runner_rolling_coefficient_num_threads=None, runner_activation_significance_floor=0.0, runner_converter_input_artifact_dir=None, runner_feature_statistics_backend='arrow', runner_logits_histogram_backend='arrow', runner_defer_component_construction=False, runner_sequence_selection_backend='columnar_gpu', runner_dashboard_output_format='auto', legacy_export_bundle_contract='auto', runner_columnar_artifact_format='parquet', runner_emit_activation_copy_rows=None, runner_overlap_batch_packaging=False, runner_sequence_top_acts_positive_only=False, runner_sequence_dedup_across_groups=False, runner_sequence_skip_dead_features=False, runner_sequence_half_open_interval_bins=False, runner_columnar_max_staged_acts_bytes=None, runner_columnar_row_chunk_size=None, runner_logits_table_mask_token_pattern=None, local_db_import_dedup_activation_rows=False, local_db_import_drop_zero_activation_rows=False, runner_prompt_bucket_schedule_file=None, runner_auto_prompt_bucket_schedule=False, runner_prompt_bucket_ceilings=<factory>, runner_prompt_bucket_scale_limit=4.0, runner_prompt_primary_acts_scale_limit=4.0, runner_prompt_batch_size_round_to=8, runner_torch_profile=False, runner_torch_profile_dir=None, runner_use_cached_activations=True, cuda_visible_devices='0', heartbeat_seconds=60, stall_timeout_seconds=0, import_to_local_db=True, allow_missing_local_db=False, local_db_url=None, local_db_import_chunk_size=65000, overlap_local_db_import=False, webapp_url='http://localhost:3000', archive_partial_dirs=True, resume_from_existing_logs=True, deduplicate_shared_prompt_tokens=True, strict_shared_prompt_count=False, prompt_bucket_ceilings=<factory>, prompt_bucket_ceiling=None, cert_bundle_path=<factory>, torch_cuda_alloc_conf='expandable_segments:True', import_only_local_db=False)[source]#
Configuration for generating, converting, and importing Neuronpedia dashboard layers.
- Parameters:
model_name (str)
model_layers (int)
sae_set (str)
neuronpedia_source_set_id (str)
neuronpedia_source_set_description (str)
creator_name (str)
release_id (str)
release_title (str)
release_url (str)
hf_weights_repo_id (str)
hf_weights_path_template (str)
hook_point (str)
prompts_huggingface_dataset_path (str)
start_layer (int)
end_layer (int)
sae_path_template (str)
prompts_dataset_mode (str)
hf_model_path (str | None)
prompts_huggingface_dataset_config_name (str | None)
prompts_huggingface_dataset_split (str | None)
prompts_dataset_text_field (str | None)
prompts_pretokenized_dataset_path (Path | None)
prompts_shared_tokens_file (Path | None)
run_root (Path)
run_name_suffix (str | None)
export_root (Path)
existing_log_path (Path | None)
pipeline_log_path (Path | None)
worker_id (str | None)
enable_layer_locks (bool)
layer_lock_stale_seconds (int)
saedashboard_repo_root (Path)
saelens_repo_root (Path)
neuronpedia_utils_root (Path)
interpretune_env_file (Path | None)
python_executable (str)
use_skip_transcoder (bool)
sae_dtype (str)
model_dtype (str)
sparsity_threshold (int)
n_prompts_total (int)
n_tokens_in_prompt (int)
n_features_per_batch (int)
n_prompts_in_forward_pass (int)
primary_acts_batch_size (int | None)
start_batch (int)
end_batch (int | None)
zero_out_bos_token (bool)
use_clt (bool)
clt_dtype (str)
clt_weights_filename (str)
dataset_streaming (bool)
model_wrapper (str)
bridge_enable_compatibility_mode (bool)
runner_log_resource_snapshots (bool)
runner_log_hook_aliases (bool)
runner_log_performance (bool)
runner_profile_rolling_substages (bool)
runner_shuffle_tokens (bool)
runner_implementation (str)
runner_cleanup_each_minibatch (bool)
runner_correlation_accumulation_device (str)
runner_rolling_coefficient_num_threads (int | None)
runner_activation_significance_floor (float)
runner_converter_input_artifact_dir (Path | None)
runner_feature_statistics_backend (str)
runner_logits_histogram_backend (str)
runner_defer_component_construction (bool)
runner_sequence_selection_backend (str)
runner_dashboard_output_format (str)
legacy_export_bundle_contract (str)
runner_columnar_artifact_format (str)
runner_emit_activation_copy_rows (bool | None)
runner_overlap_batch_packaging (bool)
runner_sequence_top_acts_positive_only (bool)
runner_sequence_dedup_across_groups (bool)
runner_sequence_skip_dead_features (bool)
runner_sequence_half_open_interval_bins (bool)
runner_columnar_max_staged_acts_bytes (int | None)
runner_columnar_row_chunk_size (int | None)
runner_logits_table_mask_token_pattern (str | None)
local_db_import_dedup_activation_rows (bool)
local_db_import_drop_zero_activation_rows (bool)
runner_prompt_bucket_schedule_file (Path | None)
runner_auto_prompt_bucket_schedule (bool)
runner_prompt_bucket_scale_limit (float)
runner_prompt_primary_acts_scale_limit (float)
runner_prompt_batch_size_round_to (int)
runner_torch_profile (bool)
runner_torch_profile_dir (Path | None)
runner_use_cached_activations (bool)
cuda_visible_devices (str | None)
heartbeat_seconds (int)
stall_timeout_seconds (int)
import_to_local_db (bool)
allow_missing_local_db (bool)
local_db_url (str | None)
local_db_import_chunk_size (int)
overlap_local_db_import (bool)
webapp_url (str)
archive_partial_dirs (bool)
resume_from_existing_logs (bool)
deduplicate_shared_prompt_tokens (bool)
strict_shared_prompt_count (bool)
prompt_bucket_ceiling (int | None)
cert_bundle_path (Path)
torch_cuda_alloc_conf (str | None)
import_only_local_db (bool)
- class interpretune.utils.NeuronpediaExplanationArtifact(feature_ref, prompt, raw_response, cleaned_explanation, artifact_path, cached_activations_path=None, import_artifact=None, database_explanation_id=None)[source]#
Result of a generated explanation artifact.
- Parameters:
feature_ref (NeuronpediaFeatureRef)
prompt (str)
raw_response (str)
cleaned_explanation (str)
artifact_path (Path)
cached_activations_path (Path | None)
import_artifact (NeuronpediaExplanationImportArtifact | None)
database_explanation_id (str | None)
- class interpretune.utils.NeuronpediaExplanationImportArtifact(export_root, explanation_batch_path, explanation_record)[source]#
Paths and record data for a Neuronpedia explanation import bundle.
- class interpretune.utils.NeuronpediaExportBundleSummary(*args, **kwargs)[source]#
Stand-in for
neuronpedia_utils.local_db_import.NeuronpediaExportBundleSummary.- Parameters:
args (Any)
kwargs (Any)
- class interpretune.utils.NeuronpediaFeatureRef(model_id, layer, index, base_url='https://www.neuronpedia.org')[source]#
Identifies a Neuronpedia feature page and its API endpoint.
- class interpretune.utils.NeuronpediaLocalExplanationCoverage(statuses, generated_artifacts, generation_failures)[source]#
Coverage summary for local explanation availability and optional backfill.
- Parameters:
statuses (list[NeuronpediaLocalExplanationStatus])
generated_artifacts (list[NeuronpediaExplanationArtifact])
generation_failures (list[NeuronpediaExplanationGenerationFailure])
- class interpretune.utils.NeuronpediaLocalExplanationStatus(feature_ref, explanation_count)[source]#
Local explanation availability for one Neuronpedia feature route.
- Parameters:
feature_ref (NeuronpediaFeatureRef)
explanation_count (int)
- class interpretune.utils.NeuronpediaLocalImportSummary(*args, **kwargs)[source]#
Stand-in for
neuronpedia_utils.local_db_import.NeuronpediaLocalImportSummary.- Parameters:
args (Any)
kwargs (Any)
- class interpretune.utils.NeuronpediaPromptInputs(tokens_after_max_activating_token, max_activating_tokens, top_positive_logits, top_activating_texts)[source]#
Derived prompt inputs for the Neuronpedia-style explanation flow.
- interpretune.utils.activation_batch_number(feature_index, batch_size=512)[source]#
Return the Neuronpedia export batch number for a feature index.
- interpretune.utils.artifact_output_path(output_dir, feature_ref)[source]#
Return the markdown artifact path using the requested timestamp-first naming scheme.
- Return type:
- Parameters:
output_dir (Path)
feature_ref (NeuronpediaFeatureRef)
- interpretune.utils.build_explanation_export_record(*, feature_ref, cleaned_explanation, artifact_path, explanation_model, cached_activations_path, author_id='clkht01d40000jv08hvalcvly', triggered_by_user_id=None, type_name='np_max-act-logits', explanation_model_name=None, explanation_cli='copilot')[source]#
Build a Neuronpedia-compatible explanation row for import or direct DB insertion.
- Return type:
- Parameters:
- interpretune.utils.build_explanation_prompt(feature_ref, prompt_inputs, *, type_name='np_max-act-logits')[source]#
Build an explanation prompt and return it with its prompt style name.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
prompt_inputs (NeuronpediaPromptInputs)
type_name (str)
- interpretune.utils.build_feature_ref(*, feature_url=None, model_id=None, layer=None, source_set=None, index=None, base_url='https://www.neuronpedia.org')[source]#
Build a feature reference from either a full feature URL or explicit metadata.
- interpretune.utils.build_np_max_act_logits_prompt(feature_ref, prompt_inputs)[source]#
Build an explanation prompt that mirrors Neuronpedia’s np-max-act-logits workflow.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
prompt_inputs (NeuronpediaPromptInputs)
- interpretune.utils.cached_activation_batch_path(feature_ref, *, cache_dir=None, batch_size=512)[source]#
Return the expected local cache path for a Neuronpedia activation batch.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
cache_dir (Path | None)
batch_size (int)
- interpretune.utils.candidate_activation_batch_sizes(batch_size=None)[source]#
Return activation batch sizes to probe, preserving order and uniqueness.
- interpretune.utils.candidate_cached_activation_batch_paths(feature_ref, *, cache_dir=None, batch_size=None)[source]#
Return candidate local cache paths for known activation batch layouts.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
cache_dir (Path | None)
batch_size (int | None)
- interpretune.utils.candidate_public_activation_batch_urls(feature_ref, *, dataset_base_url='https://neuronpedia-datasets.s3.amazonaws.com', batch_size=None)[source]#
Return candidate public activation batch URLs for known batch layouts.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
dataset_base_url (str)
batch_size (int | None)
- interpretune.utils.check_local_explanation_coverage(feature_refs, *, local_db_url=None, type_name=None)[source]#
Return local explanation counts for the provided feature refs.
- Return type:
- Parameters:
feature_refs (Iterable[NeuronpediaFeatureRef])
local_db_url (str | None)
type_name (str | None)
- interpretune.utils.check_local_neuronpedia_services(*, local_db_url=None, webapp_url='http://localhost:3000', timeout_seconds=5)[source]#
Probe the local Neuronpedia webapp and Postgres services without raising on failure.
- Return type:
- Parameters:
- interpretune.utils.clean_explanation_text(raw_response)[source]#
Extract the final explanation text from an explanation CLI response.
- interpretune.utils.collect_env_info()[source]#
Collect environmental details, logging versions of salient packages for improved reproducibility.
- Returns:
The dictionary of environmental details
- Return type:
Dict
- interpretune.utils.compare_version(package, op, version_str, use_base_version=False)[source]#
Compare package version with some requirements.
>>> compare_version("torch", operator.ge, "0.1") True >>> compare_version("does_not_exist", operator.ge, "0.0") False
- interpretune.utils.completed_layers_from_logs(*log_paths)[source]#
Parse all completed layer markers from one or more pipeline log files.
- interpretune.utils.convert_dashboard_output(config, *, layer_num, output_dir, logger=None)[source]#
Convert a SAEDashboard layer output into a Neuronpedia export bundle.
- Return type:
- Parameters:
config (NeuronpediaDashboardPipelineConfig)
layer_num (int)
output_dir (Path)
logger (Logger | None)
- interpretune.utils.default_np_cache_dir()[source]#
Return the local Interpretune Neuronpedia cache directory.
- Return type:
- interpretune.utils.derive_np_max_act_logits_inputs(feature_payload, *, top_activations_limit=10, tokens_around_max=24)[source]#
Reconstruct the four Neuronpedia input lists used by np-max-act-logits.
- interpretune.utils.ensure_cached_activation_batch(feature_ref, *, cache_dir=None, timeout_seconds=60, dataset_base_url='https://neuronpedia-datasets.s3.amazonaws.com', batch_size=512)[source]#
Ensure the activation batch for a feature exists in the local cache.
- interpretune.utils.ensure_local_feature_explanations(feature_refs, *, generate_missing=False, regenerate_existing=False, output_dir=PosixPath('/tmp/generated_np_explanations'), explanation_model=None, timeout_seconds=120, cache_dir=None, local_db_url=None, explanation_author_id='clkht01d40000jv08hvalcvly', triggered_by_user_id=None, type_name='np_max-act-logits', explanation_model_name=None, max_retries=2, retry_backoff_seconds=5.0, cli_spec=None)[source]#
Check local explanation coverage and optionally backfill missing entries.
regenerate_existingre-generates features that ALREADY have a local explanation instead of skipping them. Nothing is deleted – the new explanation is inserted alongside, so the operation is safe to repeat. Use it to prove the generation path actually ran end to end on a database that is already populated, where every feature would otherwise be skipped and the run would report full coverage without calling the CLI once. Requiresgenerate_missing=True.- Return type:
- Parameters:
feature_refs (Iterable[NeuronpediaFeatureRef])
generate_missing (bool)
regenerate_existing (bool)
output_dir (Path)
explanation_model (str | None)
timeout_seconds (int)
cache_dir (Path | None)
local_db_url (str | None)
explanation_author_id (str)
triggered_by_user_id (str | None)
type_name (str)
explanation_model_name (str | None)
max_retries (int)
retry_backoff_seconds (float)
cli_spec (ExplanationCliSpec | None)
- interpretune.utils.feature_tuples_to_feature_refs(*, model_id, source_set, feature_tuples, base_url='https://www.neuronpedia.org')[source]#
Convert (layer, feature_index) tuples into Neuronpedia feature refs.
- interpretune.utils.fetch_feature_payload(feature_ref, timeout_seconds=30)[source]#
Fetch the full feature payload from Neuronpedia’s public feature API.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
timeout_seconds (int)
- interpretune.utils.generate_explanation_artifact(*, feature_ref, output_dir=PosixPath('/tmp/generated_np_explanations'), explanation_model=None, timeout_seconds=120, cache_dir=None, write_neuronpedia_import_data=False, insert_into_local_db=False, local_db_url=None, explanation_author_id='clkht01d40000jv08hvalcvly', triggered_by_user_id=None, type_name='np_max-act-logits', explanation_model_name=None, max_retries=2, retry_backoff_seconds=5.0, cli_spec=None)[source]#
Generate a Neuronpedia-style explanation artifact using cached activations and an explanation CLI.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
output_dir (Path)
explanation_model (str | None)
timeout_seconds (int)
cache_dir (Path | None)
write_neuronpedia_import_data (bool)
insert_into_local_db (bool)
local_db_url (str | None)
explanation_author_id (str)
triggered_by_user_id (str | None)
type_name (str)
explanation_model_name (str | None)
max_retries (int)
retry_backoff_seconds (float)
cli_spec (ExplanationCliSpec | None)
- interpretune.utils.import_neuronpedia_export_bundle_local_db(export_root, *, local_db_url=None, prefer_arrow_for_tables=(), prefer_copy_for_tables=(), artifact_format_by_table=None)[source]#
Import a Neuronpedia export bundle through the Neuronpedia-owned importer.
- interpretune.utils.insert_explanation_record_local_db(explanation_record, *, local_db_url)[source]#
Insert a generated explanation row into a local Neuronpedia Postgres database.
- interpretune.utils.instantiate_class(init, args=None, import_only=False)[source]#
Instantiates a class with the given args and init. Accepts class definitions with a “class_path”.
- interpretune.utils.invoke_explanation_cli_with_retries(prompt, *, explanation_model=None, timeout_seconds=120, max_retries=2, retry_backoff_seconds=5.0, cli_spec=None)[source]#
Invoke the explanation CLI and retry transient timeout failures with exponential backoff.
- interpretune.utils.load_activation_batch_records(batch_path)[source]#
Load a cached activation batch file exported by Neuronpedia.
- interpretune.utils.load_cached_feature_activations(feature_ref, *, cache_dir=None, timeout_seconds=60)[source]#
Load cached activation rows for one feature from the public export batch.
- interpretune.utils.load_feature_payload_with_cached_activations(feature_ref, *, cache_dir=None, timeout_seconds=60)[source]#
Fetch feature metadata from the API and prefer cached export activations when available.
- interpretune.utils.module_available(module_path)[source]#
Check if a module path is available in your environment.
>>> module_available('os') True >>> module_available('os.bla') False >>> module_available('bla.bla') False
- interpretune.utils.move_data_to_device(batch, device)[source]#
Transfers a collection of data to the given device. Any object that defines a method
to(device)will be moved and all other objects in the collection will be left untouched.- Parameters:
- Return type:
- Returns:
the same collection but with all contained tensors residing on the new device.
See also
- interpretune.utils.package_available(package_name)[source]#
Check if a package is available in your environment.
>>> package_available('os') True >>> package_available('bla') False
- interpretune.utils.parse_feature_url(feature_url)[source]#
Parse a Neuronpedia feature page URL or feature API URL into a feature reference.
- Return type:
- Parameters:
feature_url (str)
- interpretune.utils.public_activation_batch_url(feature_ref, *, dataset_base_url='https://neuronpedia-datasets.s3.amazonaws.com', batch_size=512)[source]#
Return the public S3 URL for a feature’s activation batch export.
- Return type:
- Parameters:
feature_ref (NeuronpediaFeatureRef)
dataset_base_url (str)
batch_size (int)
- interpretune.utils.rank_zero_debug(*args, stacklevel=4, **kwargs)[source]#
Emit debug-level messages only on global rank 0.
- interpretune.utils.rank_zero_deprecation(message, stacklevel=5, **kwargs)[source]#
Emit a deprecation warning only on global rank 0.
- interpretune.utils.rank_zero_info(*args, stacklevel=4, **kwargs)[source]#
Emit info-level messages only on global rank 0.
- interpretune.utils.rank_zero_only(fn, default=None)[source]#
Wrap a function to call internal function only in rank zero.
Function that can be used as a decorator to enable a function/method being called only on global rank 0.
- interpretune.utils.rank_zero_warn(message, stacklevel=4, **kwargs)[source]#
Emit warn-level messages only on global rank 0.
- interpretune.utils.resolve_local_neuronpedia_db_url(local_db_url=None, *, env=None)[source]#
Resolve the best local Neuronpedia Postgres URL from explicit input or environment.
- interpretune.utils.rewrite_container_postgres_url_for_host(connection_url, *, env=None, container_hostname='postgres', host='127.0.0.1', port_env_var='POSTGRES_HOST_PORT')[source]#
Rewrite a docker-only Postgres URL to the host-mapped local port when configured.
- interpretune.utils.run_dashboard_pipeline(config)[source]#
Run dashboard generation, conversion, and optional local import for a layer range.
- Return type:
- Parameters:
config (NeuronpediaDashboardPipelineConfig)
- interpretune.utils.to_device(device, obj)[source]#
Move a
torch.nn.Moduleor a collection of tensors to the current device, if it is not already on that device.- Parameters:
- Return type:
- Returns:
A reference to the object that was moved to the new device.
- interpretune.utils.write_explanation_import_bundle(*, output_dir, feature_ref, explanation_record)[source]#
Write a Neuronpedia explanation import bundle under a source-root layout.
- Return type:
- Parameters:
output_dir (Path)
feature_ref (NeuronpediaFeatureRef)