Add forward and backward transfer metrics

.claude/skills/custom-experiment/SKILL.md

@@ -85,6 +85,8 @@ poetry run python -m src.main --config $2
 ```
 Report drift events, final accuracy, and the output CSV path (the config's `visualization.input`). Note the package emits this CSV for inspection; it does not ship a built-in dashboard renderer.
 
+The CSV also carries `eval/fwt` and `eval/bwt` per drift event — the gain adapting made on the triggering window, and how far past tasks moved since they were learned. Both come free from `BaseModelHarness`, so a custom harness gets them without extra work; sign follows the metric's direction, so for a lower-is-better metric positive `bwt` means forgetting. See `docs/tracking.md` "Transfer Metrics".
+
 ## Notes
 - Registration here uses the in-repo factory pattern. To instead drive apeiron from your *own* project without editing this repo, use the integrate-apeiron skill.
 - The repo also has older `new-harness` / `new-config` skills covering pieces of this; they are stale (pre-`src/apeiron/` layout) and slated for refresh — prefer this skill.

.claude/skills/explore-examples/SKILL.md

@@ -57,6 +57,7 @@ The config default is `wandb`. Before running, ask the user to choose, and pass
 ### 5. Report results
 - Summarize from the run output: whether drift was detected and how many times, final accuracy, and the output CSV path (the config's `visualization.input`).
 - The package emits this CSV for inspection; it does not ship a built-in dashboard renderer, so point the user at the CSV for further plotting.
+- The CSV carries one row per drift event for `eval/fwt` (what adapting gained on the triggering window) and `eval/bwt` (how far past tasks moved since they were learned; absent on the first event). They are the quickest read on whether adaptation is trading history away — worth quoting alongside `eval/test_curr_acc` and `eval/test_hist_acc`. Sign follows the metric's direction, so for accuracy examples negative `bwt` means forgetting. See `docs/tracking.md` "Transfer Metrics".
 
 ## Notes
 - Quick first run, copy-paste safe: `poetry run python -m src.main --config examples/mnist/mnist.toml --set logging.backend=none`

CLAUDE.md

@@ -30,7 +30,7 @@ The installable package lives under `src/apeiron/` (imported as `apeiron`; see `
 
 ### Core Pipeline
 1. **Config** (`src/apeiron/config/configuration.py`): TOML-based config parsed into frozen dataclasses (`Config`, `ModelCfg`, `DataCfg`, `TrainCfg`, `ContinualLearningCfg`, `DriftDetectionCfg`, `LoggingCfg`, `VisualizationCfg`). Supports `--set key=val` CLI overrides and `APP_` env var overrides.
-2. **Model Harness** (`src/apeiron/model/torch_model_harness.py`): Abstract `BaseModelHarness` providing `get_stream_dataloader()`, `get_train_dataloaders()`, `get_hist_dataloaders()`, `update_data_stream()`, `get_criterion()`, `get_optmizer()`, and `eval_metrics` dict.
+2. **Model Harness** (`src/apeiron/model/torch_model_harness.py`): Abstract `BaseModelHarness` providing `get_stream_dataloader()`, `get_train_dataloaders()`, `get_hist_dataloaders()`, `update_data_stream()`, `get_criterion()`, `get_optmizer()`, and `eval_metrics` dict. Also keeps a per-task registry for transfer metrics -- `register_task()`, `eval_past_tasks()`, `task_diagonals` -- which subclasses inherit unchanged (see `docs/tracking.md` "Transfer Metrics").
 3. **Driver** (`src/apeiron/driver/continuous_monitor.py`): `ContinuousMonitor` orchestrates the monitoring loop -- evaluates batches, checks drift at intervals, dispatches CL training on drift.
 4. **Drift Detection** (`src/apeiron/drift_detection/`): `BaseDriftDetector` ABC with `update(value) -> DriftSignal`. Implementations: ADWINDetector, KSWINDetector, PageHinkleyDetector, ModelPerformanceDetector, ModelEvalDetector, EnsembleDetector.
 5. **Training** (`src/apeiron/training/continuous_trainer.py`): `ContinuousTrainer` runs outer/inner CL loops with gradient accumulation.

docs/configurations.md

@@ -254,10 +254,13 @@ drift/regime
 drift/score
 drift/step
 eval/accuracy
+eval/bwt
+eval/fwt
 eval/loss
 eval/step
 eval/test_curr_acc
 eval/test_hist_acc
+eval/test_pre_cl_acc
 ```
 
 Example output file:

docs/tracking.md

@@ -75,6 +75,9 @@ metrics below, each stage also emits its own step counter (`eval/step`,
 | `eval/accuracy`, `eval/loss` | Per-batch metrics from the harness's `eval_metrics` dict. This is the raw monitoring signal, logged every stream batch. |
 | `eval/test_curr_acc` | Accuracy on the *current* regime, logged once after each CL round completes. |
 | `eval/test_hist_acc` | Accuracy on the *historical* regime, same cadence. Absent when the harness supplies no historical loaders. |
+| `eval/test_pre_cl_acc` | Accuracy on the current regime measured *before* the CL round ran, same cadence. |
+| `eval/fwt` | Forward transfer: what adapting to this window gained on it. See [Transfer Metrics](#transfer-metrics). |
+| `eval/bwt` | Backward transfer: how much past tasks moved since they were learned. Absent on the first drift event. |
 
 **`drift/`** — the detector, logged once per `detection_interval`.
 
@@ -101,6 +104,61 @@ The whole `Config` is recorded as run config/params, so `update_mode`,
 `detector_name`, and the detector hyperparameters are all filterable and
 groupable in the runs table.
 
+## Transfer Metrics
+
+`eval/fwt` and `eval/bwt` are indexed by **tasks**, where a task is one drift
+ event — the window the detector fired on and the CL loop adapted to — so `T`
+ counts adaptations, not stream windows.
+
+They are entries of the train-test matrix `R`, where `R[i][j]` is the score on
+task `j` after the model finished learning task `i`:
+
+| Metric | Definition | Reads |
+|---|---|---|
+| `eval/fwt` | `R[i][i] - R[i-1][i]` | The current window scored after adapting, minus its score before. The gain CL delivered on the task that triggered it. Logged at every drift event, the first included. |
+| `eval/bwt` | `(1/(T-1)) * sum over i<T of ( R[T][i] - R[i][i] )` | Each past task scored now, minus its score right after it was learned. This compares one task across two model states, which is what makes it forgetting rather than a difficulty gap. Absent on the first drift event, where the sum is empty. |
+
+`R[i-1][i]` is logged directly as `eval/test_pre_cl_acc` and `R[i][i]` as
+`eval/test_curr_acc`, so `fwt` is reconstructible from the CSV. The
+below-diagonal cells come from a per-task registry the harness maintains: after
+each CL round, `BaseModelHarness.register_task()` freezes that window's
+validation split into a standalone eval set paired with its `R[i][i]`, and
+`eval_past_tasks()` replays the current model over all of them.
+
+### Reading the Sign
+
+Both metrics are raw differences of `R`, so **the sign of the difference inherits
+the direction of the metric**. For a classification harness `R` holds an accuracy
+and bigger is better; for a regression harness it holds an error and bigger is
+worse. Every reading therefore flips between the two:
+
+| | classification (accuracy, higher better) | regression (MAE/MSE, lower better) |
+|---|---|---|
+| `fwt` > 0 | adapting raised accuracy on the window — CL helped | adapting raised error — **CL hurt** |
+| `fwt` < 0 | CL hurt | **CL helped** |
+| `bwt` > 0 | past tasks score better than when learned — backward transfer | past-task error grew — **forgetting** |
+| `bwt` < 0 | **forgetting** | past-task error shrank — backward transfer |
+
+Check the harness's `higher_is_better` before comparing runs across examples.
+
+One trap on the accuracy side: positive `bwt` means past tasks improved relative
+to their own diagonal, which can happen because real backward transfer occurred
+*or* because `R[i][i]` was weak to begin with. A short `train.max_iter` leaves the
+diagonal undertrained and will manufacture positive `bwt` that means nothing.
+Sanity-check `eval/test_curr_acc` at each event before reading a positive value as
+transfer.
+
+Two limits worth knowing. `BaseModelHarness.max_task_records` (default 50) caps
+the registry; past that, `bwt` averages over the retained tasks rather than all
+`T-1`. And `eval/bwt` differs from `eval/test_hist_acc` on purpose —
+`test_hist_acc` is a single *pooled* evaluation over the concatenated history and
+carries no reference point, so it conflates forgetting with windows that were
+simply harder.
+
+This `fwt` is the CL gain on the triggering task. It is not the Lopez-Paz &
+Ranzato form `R[i-1][i] - b_i`, which measures zero-shot transfer against a
+baseline model `b_i`; no baseline term is computed here.
+
 ## Reading the Charts
 
 Both figures below are the W&B workspace view on the MNIST example: runs logged
@@ -159,6 +217,9 @@ The green-versus-blue contrast is the argument for replay in one image, and it i
 the contrast the live-stream figure cannot show. It is also what
 `eval/test_hist_acc` measures continuously during a normal run — see
 `mix_historic_data` in [`continuous_learning.md`](continuous_learning.md).
+`eval/bwt` puts a number on the same effect per drift event, against each task's
+own starting point rather than a pooled average — see
+[Transfer Metrics](#transfer-metrics).
 
 ### Cross-Referencing Other Stages
 

src/apeiron/model/torch_model_harness.py

@@ -5,7 +5,7 @@
 
 import torch
 from torch import nn, Tensor
-from torch.utils.data import DataLoader
+from torch.utils.data import DataLoader, TensorDataset
 from torch.optim import Optimizer
 
 from apeiron.config.configuration import Config
@@ -34,6 +34,11 @@ def __init__(self, cfg: Config, model: nn.Module):
 
         self.eval_metrics: Dict[str, MetricFn] = {}
 
+        # One entry per drift event, oldest first: the frozen validation split of
+        # the window that was adapted to, paired with R[i][i] (see register_task).
+        self._task_records: List[Tuple[DataLoader, List[float]]] = []
+        self.max_task_records: int = 50
+
     @abstractmethod
     def get_optmizer(self) -> Optimizer:
         """
@@ -104,13 +109,13 @@ def _to_scalar(x: Tensor | float) -> float:
         return float(x)
 
     @torch.no_grad()
-    def eval(self) -> List[float]:
+    def _eval_loader(self, loader: DataLoader) -> List[float]:
         """Stream over batches; return mean(metric) over batches (order preserved)."""
         self.model.eval()
         sums = [0.0 for _ in self.eval_metrics]
         counts = [0 for _ in self.eval_metrics]
 
-        for batch in self.get_train_dataloaders()[1]:  # assumes iterable
+        for batch in loader:  # assumes iterable
             x, y = self._unpack(batch)
             x, y = x.to(self.cfg.device), y.to(self.cfg.device)
 
@@ -140,6 +145,11 @@ def eval(self) -> List[float]:
 
         return [s / c for s, c in zip(sums, counts)]
 
+    @torch.no_grad()
+    def eval(self) -> List[float]:
+        """Stream over batches; return mean(metric) over batches (order preserved)."""
+        return self._eval_loader(self.get_train_dataloaders()[1])
+
     @torch.no_grad()
     def history_eval(self) -> Optional[List[float]]:
         """Stream over batches; return mean(metric) over batches (order preserved).
@@ -150,39 +160,63 @@ def history_eval(self) -> Optional[List[float]]:
         if hist_loaders is None or hist_loaders[1] is None:
             return None
 
-        self.model.eval()
-        sums = [0.0 for _ in self.eval_metrics]
-        counts = [0 for _ in self.eval_metrics]
+        return self._eval_loader(hist_loaders[1])
+
+    # ----- per-task evaluation (train-test matrix R) -----
+
+    def register_task(self, diagonal_metrics: List[float]) -> None:
+        """Record the task just finished so later events can measure forgetting.
 
-        for batch in hist_loaders[1]:
+        A *task* is one drift event: the window the detector fired on and the CL
+        loop adapted to. This freezes that window's validation split into a
+        standalone eval set and stores it alongside ``diagonal_metrics`` --
+        ``R[i][i]``, the score on the window measured right after adapting to it.
+        Both travel in the same record so eviction can never misalign a task's
+        eval set from its diagonal.
+
+        The split is copied into memory rather than referenced: the harness drops
+        its window tensors as the stream advances, and a plain ``DataLoader``
+        reference would keep the whole window alive through a view.
+
+        :param diagonal_metrics: ``eval()`` output for the current window, taken
+            after the CL loop finished.
+        :type diagonal_metrics: List[float]
+        """
+        xs: List[Tensor] = []
+        ys: List[Tensor] = []
+        for batch in self.get_train_dataloaders()[1]:
             x, y = self._unpack(batch)
-            x, y = x.to(self.cfg.device), y.to(self.cfg.device)
+            xs.append(x.detach().cpu().clone())
+            ys.append(y.detach().cpu().clone())
 
-            # TODO: Add cuda amp support later. Needs config entry for amp
-            # if self.cfg.amp:
+        frozen = DataLoader(
+            TensorDataset(torch.cat(xs), torch.cat(ys)),
+            batch_size=self.cfg.train.batch_size,
+            shuffle=False,
+        )
+        self._task_records.append((frozen, list(diagonal_metrics)))
 
-            #     with torch.autocast(
-            #         device_type=self.device.type,
-            #         dtype=(
-            #             torch.float16 if self.device.type == "cuda" else torch.bfloat16
-            #         ),
-            #     ):
-            #         y_hat = self.model(x)
-            # else:
-            y_hat = self.model(x)
+        # Cap retained tasks; BWT then averages over the surviving ones.
+        while len(self._task_records) > self.max_task_records:
+            self._task_records.pop(0)
 
-            batch_size = y.shape[0]
-            for i, m in enumerate(self.eval_metrics.values()):
-                metric_value = self._to_scalar(m(y_hat, y))
-                # For metrics that return percentages (like accuracy), we need to
-                # convert back to counts for proper averaging across variable batch sizes
-                sums[i] += metric_value * batch_size
-                counts[i] += batch_size
+    @torch.no_grad()
+    def eval_past_tasks(self) -> List[List[float]]:
+        """Score the current model on every registered task's frozen eval set.
 
-        if counts[0] == 0:
-            raise RuntimeError("Empty loader: nothing to evaluate.")
+        Returns row ``T`` of the train-test matrix below the diagonal --
+        ``[R[T][i] for i < T]``, oldest task first, index-aligned with
+        :attr:`task_diagonals`. Empty until at least one task is registered.
+        """
+        return [self._eval_loader(loader) for loader, _ in self._task_records]
 
-        return [s / c for s, c in zip(sums, counts)]
+    @property
+    def task_diagonals(self) -> List[List[float]]:
+        """``R[i][i]`` per registered task, oldest first.
+
+        Index-aligned with :meth:`eval_past_tasks`.
+        """
+        return [diagonal for _, diagonal in self._task_records]
 
     @property
     def ckpts_enabled(self) -> bool:

src/apeiron/training/continuous_trainer.py

@@ -83,6 +83,41 @@ def _log_validation(
         # increment=False: annotate the CL round's step, do not advance it.
         logger.log(payload, commit=False, increment=False)
 
+    def compute_bwt(self, metric_index: int = 0) -> Optional[float]:
+        """Backward transfer for the task that just finished adapting.
+
+        ``BWT = (1/(T-1)) * sum_{i<T} ( R[T][i] - R[i][i] )``
+
+        where a *task* is one drift event, ``R[T][i]`` is the current model's
+        score on task ``i``'s validation split and ``R[i][i]`` is the score on
+        that same split recorded right after adapting to it. It therefore
+        compares one task across two model states -- the definition of
+        forgetting -- rather than comparing different tasks at one state.
+
+        Sign follows the raw metric, so the reading depends on the metric's
+        direction: with a higher-is-better metric (accuracy) negative means
+        forgetting, while with a lower-is-better one (SLAC-FEL's MAE) *positive*
+        means forgetting.
+
+        :param metric_index: which entry of ``eval_metrics`` to use, matching the
+            index behind ``test_curr_acc``/``test_hist_acc``.
+        :type metric_index: int
+
+        :return: BWT over the retained past tasks, or None before any task has
+            been registered (the first drift event, where the sum is empty).
+        :rtype: Optional[float]
+        """
+        past_metrics = self.modelHarness.eval_past_tasks()
+        if not past_metrics:
+            return None
+
+        diagonals = self.modelHarness.task_diagonals
+        deltas = [
+            row[metric_index] - diagonal[metric_index]
+            for row, diagonal in zip(past_metrics, diagonals)
+        ]
+        return sum(deltas) / len(deltas)
+
     def outer_cl_training_loop(
         self,
         drift_event_id: int = 0,
@@ -101,9 +136,9 @@ def outer_cl_training_loop(
         cur_validation_metrics = self.modelHarness.eval()
         hist_validation_metrics = self.modelHarness.history_eval()
 
-        self._log_validation(
-            "pre", cur_validation_metrics, hist_validation_metrics, drift_event_id
-        )
+        # R[i-1][i]: this window scored by the model that has not yet adapted to
+        # it. Kept for the FWT delta once the post-CL score (R[i][i]) is in.
+        pre_cl_validation_metrics = cur_validation_metrics
 
         logger.info("==== Continual Learning ====")
         logger.info("\tInitial test acc: {}".format(cur_validation_metrics[0]), level=1)
@@ -165,22 +200,31 @@ def outer_cl_training_loop(
         else:
             logger.info("\tNo historical data available for evaluation", level=1)
 
+        # FWT = R[i][i] - R[i-1][i]: how much adapting to this window moved the
+        # score on it, i.e. the gain CL delivered on the task that triggered.
+        # Available at every drift event, including the first.
+        fwt = cur_validation_metrics[0] - pre_cl_validation_metrics[0]
+        logger.info(f"\tFWT: {fwt:.4g}", level=1)
+
+        bwt = self.compute_bwt()
+        if bwt is not None:
+            logger.info(f"\tBWT: {bwt:.4g}", level=1)
+
         logger.stage("eval")
+        eval_metrics: dict[str, float] = {
+            "test_curr_acc": cur_validation_metrics[0],
+            "test_pre_cl_acc": pre_cl_validation_metrics[0],
+            "fwt": fwt,
+        }
         if hist_validation_metrics is not None:
-            logger.log(
-                {
-                    "test_curr_acc": cur_validation_metrics[0],
-                    "test_hist_acc": hist_validation_metrics[0],
-                },
-                commit=False,
-            )
-        else:
-            logger.log(
-                {
-                    "test_curr_acc": cur_validation_metrics[0],
-                },
-                commit=False,
-            )
+            eval_metrics["test_hist_acc"] = hist_validation_metrics[0]
+        if bwt is not None:
+            eval_metrics["bwt"] = bwt
+        logger.log(eval_metrics, commit=False)
+
+        # Register *after* BWT so this event's window becomes task T only for
+        # subsequent events -- R[T][T] belongs on the diagonal, not in the sum.
+        self.modelHarness.register_task(cur_validation_metrics)
 
         if self.profiler:
             flops_perf = self.profiler.get_performance()