👉 TL;DR. Sparse autoencoders can reconstruct identical representations while their features depend on different inputs. We introduce optimal input scope curves to measure how compactly controlled inputs explain SAE feature activations.
📄 arXiv paper coming soon | 🎞️ 5-minute tutorial | 🧪 Demo
Figure 1. Two bijective input maps provide exactly the same representation samples to the same frozen SAE. Yet the inputs needed to explain its feature activations differ. Across five learned SAEs, the budget needed to explain 90% of activation variance increases by 41% and 49% under the mixed maps.
💻 Click to open the tutorial in your browser
- 🧩 Identical representations, different dependencies. The same SAE and representation distribution can require exactly r times the input scope budget under an order-r input map, even when sparse coding uniquely selects the shared dictionary.
- 📐 Optimal scope curves. The curve finds the smallest feature-activation variance left unexplained under a total input-dependency budget, distinguishing sparsity from localization.
- 📈 Learned SAEs confirm the separation. Across five trained overcomplete SAEs, the mean budget for 90% explained activation variance rises from 136.0 to 192.4 and 202.8 under three- and five-factor XOR maps.
- 🎲 Paired interventions recover the curve. The mean maximum normalized curve error drops from 0.123 to 0.0109 as pairs per candidate scope increase from 32 to 2,048.
- 🛡️ Lower bounds on localization cost. Compatibility and spectral results connect joint input effects, decoder directions, and the number of input-feature dependencies.
- ⚡ A lightweight estimator. The core implementation uses NumPy. The included five-SAE demo runs directly from released activations, with no model download or retraining.
🧩 Input scopes and SAE features
For controlled inputs
SAE sparsity counts active features;
📉 Optimal input scope curve
For a proposed scope
The optimal scope curve and its inverse are
🎲 Estimation from paired interventions
For independent
Paired resampling estimates the omission error for a candidate scope. Combining those estimates with scope optimization recovers an empirical curve; the paper also provides simultaneous finite-sample control after scope selection.
🛡️ Compatibility and spectral bounds
Pure input interactions require features whose scopes contain the jointly contributing inputs. Those features must also provide decoder directions capable of representing the interaction. The paper derives lower bounds from these two constraints and from the spectrum of higher-order input effects.
The controlled experiment fixes the representation samples and trained SAE, then changes their correspondence with the input variables.
| Measurement | What it describes | Identity vs. XOR maps |
|---|---|---|
| Reconstruction error | Accuracy of the reconstructed representation | Same |
| Activation sparsity | Number of active SAE features | Same |
| Representation and activation samples | Distribution of |
Same |
| Optimal input scope curve | Inputs required to explain SAE activations | Different |
| Paired input interventions | Activation variance omitted by a chosen scope | Different |
The results separate representation quality from input localization: the same sparse features can have different input-dependency structures.
git clone https://github.com/OpenSAE/SAE-Input-Localization.git
cd SAE-Input-Localization
pip install -e .The core methods need only NumPy (Python 3.10+). For tests, use pip install -e ".[dev]"; for training experiments, use pip install -e ".[experiments]".
import numpy as np
from sae_localize import exact_scope_curve
# Two SAE features over three controlled binary inputs.
x = np.arange(8)
z = np.column_stack([x & 1, (x >> 1) & 1]).astype(float)
# A bijection mixes the first two inputs, preserving every SAE code sample.
mixed_ids = ((x & 1) ^ ((x >> 1) & 1)) | (x & ~1)
identity = exact_scope_curve(z, n_factors=3)
mixed = exact_scope_curve(z[mixed_ids], n_factors=3)
print(identity.budget(0), mixed.budget(0))
# 2 3💡 The representation and activation samples are unchanged. Under the mixed input map, one feature needs both input factors instead of just one, increasing the exact dependency budget from 2 to 3.
🧰 Your own SAE. Supply a NumPy array of frozen feature activations with shape (2**n_factors, n_features), ordered by the binary input configuration. Compute its curve with exact_scope_curve(z, n_factors=n) or run:
python demo/custom_activations.py my_codes.npy --n-factors 8🧠 Figure 2. Exact separation for the same frozen SAE
For
📈 Figure 3. Learned SAE localization and finite-sample recovery
Five independently trained representation–SAE pairs are evaluated over all 256 configurations of eight controlled binary inputs.
| Input map | Mean budget at 90% explained variance | Change |
|---|---|---|
| Identity | 136.0 | — |
| XOR, arity 3 | 192.4 | +41.5% |
| XOR, arity 5 | 202.8 | +49.1% |
Finite paired sampling progressively recovers the exact curves:
| Pairs per candidate scope | 32 | 128 | 512 | 2,048 |
|---|---|---|---|---|
| Mean maximum normalized curve error | 0.123 | 0.0472 | 0.0216 | 0.0109 |
🛡️ Appendix Figure 4. Spectral and feature-order bounds
| Input map | Exact budget | Spectral bound | Feature-order bound |
|---|---|---|---|
| Identity | 136.0 | 4.2 | 71.2 |
| XOR, arity 3 | 192.4 | 12.6 | 145.0 |
| XOR, arity 5 | 202.8 | 20.6 | 150.2 |
The feature-order bound is tighter because it retains SAE feature-level interaction information discarded by the representation-level spectral bound.
Run the five-seed learned-SAE comparison using the included reference activations:
python demo/demo.pyMap Five seed budgets Mean ± sample SD
Identity [128, 147, 168, 129, 108] 136.0 ± 22.6
XOR arity 3 [206, 197, 197, 186, 176] 192.4 ± 11.6
XOR arity 5 [215, 204, 205, 192, 198] 202.8 ± 8.6
✅ The demo recomputes the optimized scope curves and reproduces the headline results. To retrain all five SAEs or regenerate the figures, see the reproduction guide.
| Figure | Main finding | Reproduction |
|---|---|---|
| Figure 1 | Same representations and SAE, different input dependence | Overview |
| Figure 2 | Exact separation across interaction orders | verify_family.py |
| Figure 3 | Learned SAE scope curves and paired recovery | pilot.py, audit.py |
| Appendix Figure 4 | Spectral and feature-order bounds | code_order_bound.py |
Part of OpenSAE, alongside Behavioral Compositionality (SAR), Representation Redistribution (RSS), and the SAEBase research infrastructure.




