diff --git a/.claude/skills/custom-experiment/SKILL.md b/.claude/skills/custom-experiment/SKILL.md index 39bb6ea..3a49eec 100644 --- a/.claude/skills/custom-experiment/SKILL.md +++ b/.claude/skills/custom-experiment/SKILL.md @@ -85,6 +85,8 @@ poetry run python -m src.main --config $2 ``` Report drift events, final accuracy, and the output CSV path (the config's `visualization.input`). Note the package emits this CSV for inspection; it does not ship a built-in dashboard renderer. +The CSV also carries `eval/fwt` and `eval/bwt` per drift event — the gain adapting made on the triggering window, and how far past tasks moved since they were learned. Both come free from `BaseModelHarness`, so a custom harness gets them without extra work; sign follows the metric's direction, so for a lower-is-better metric positive `bwt` means forgetting. See `docs/tracking.md` "Transfer Metrics". + ## Notes - Registration here uses the in-repo factory pattern. To instead drive apeiron from your *own* project without editing this repo, use the integrate-apeiron skill. - The repo also has older `new-harness` / `new-config` skills covering pieces of this; they are stale (pre-`src/apeiron/` layout) and slated for refresh — prefer this skill. diff --git a/.claude/skills/explore-examples/SKILL.md b/.claude/skills/explore-examples/SKILL.md index 5189e62..3f14f4c 100644 --- a/.claude/skills/explore-examples/SKILL.md +++ b/.claude/skills/explore-examples/SKILL.md @@ -57,6 +57,7 @@ The config default is `wandb`. Before running, ask the user to choose, and pass ### 5. Report results - Summarize from the run output: whether drift was detected and how many times, final accuracy, and the output CSV path (the config's `visualization.input`). - The package emits this CSV for inspection; it does not ship a built-in dashboard renderer, so point the user at the CSV for further plotting. +- The CSV carries one row per drift event for `eval/fwt` (what adapting gained on the triggering window) and `eval/bwt` (how far past tasks moved since they were learned; absent on the first event). They are the quickest read on whether adaptation is trading history away — worth quoting alongside `eval/test_curr_acc` and `eval/test_hist_acc`. Sign follows the metric's direction, so for accuracy examples negative `bwt` means forgetting. See `docs/tracking.md` "Transfer Metrics". ## Notes - Quick first run, copy-paste safe: `poetry run python -m src.main --config examples/mnist/mnist.toml --set logging.backend=none` diff --git a/CLAUDE.md b/CLAUDE.md index 623750e..de5c55f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,7 @@ The installable package lives under `src/apeiron/` (imported as `apeiron`; see ` ### Core Pipeline 1. **Config** (`src/apeiron/config/configuration.py`): TOML-based config parsed into frozen dataclasses (`Config`, `ModelCfg`, `DataCfg`, `TrainCfg`, `ContinualLearningCfg`, `DriftDetectionCfg`, `LoggingCfg`, `VisualizationCfg`). Supports `--set key=val` CLI overrides and `APP_` env var overrides. -2. **Model Harness** (`src/apeiron/model/torch_model_harness.py`): Abstract `BaseModelHarness` providing `get_stream_dataloader()`, `get_train_dataloaders()`, `get_hist_dataloaders()`, `update_data_stream()`, `get_criterion()`, `get_optmizer()`, and `eval_metrics` dict. +2. **Model Harness** (`src/apeiron/model/torch_model_harness.py`): Abstract `BaseModelHarness` providing `get_stream_dataloader()`, `get_train_dataloaders()`, `get_hist_dataloaders()`, `update_data_stream()`, `get_criterion()`, `get_optmizer()`, and `eval_metrics` dict. Also keeps a per-task registry for transfer metrics -- `register_task()`, `eval_past_tasks()`, `task_diagonals` -- which subclasses inherit unchanged (see `docs/tracking.md` "Transfer Metrics"). 3. **Driver** (`src/apeiron/driver/continuous_monitor.py`): `ContinuousMonitor` orchestrates the monitoring loop -- evaluates batches, checks drift at intervals, dispatches CL training on drift. 4. **Drift Detection** (`src/apeiron/drift_detection/`): `BaseDriftDetector` ABC with `update(value) -> DriftSignal`. Implementations: ADWINDetector, KSWINDetector, PageHinkleyDetector, ModelPerformanceDetector, ModelEvalDetector, EnsembleDetector. 5. **Training** (`src/apeiron/training/continuous_trainer.py`): `ContinuousTrainer` runs outer/inner CL loops with gradient accumulation. diff --git a/docs/configurations.md b/docs/configurations.md index 7bacdc0..32ecd8e 100644 --- a/docs/configurations.md +++ b/docs/configurations.md @@ -254,10 +254,13 @@ drift/regime drift/score drift/step eval/accuracy +eval/bwt +eval/fwt eval/loss eval/step eval/test_curr_acc eval/test_hist_acc +eval/test_pre_cl_acc ``` Example output file: diff --git a/docs/tracking.md b/docs/tracking.md index b89bb2b..426d1a4 100644 --- a/docs/tracking.md +++ b/docs/tracking.md @@ -75,6 +75,9 @@ metrics below, each stage also emits its own step counter (`eval/step`, | `eval/accuracy`, `eval/loss` | Per-batch metrics from the harness's `eval_metrics` dict. This is the raw monitoring signal, logged every stream batch. | | `eval/test_curr_acc` | Accuracy on the *current* regime, logged once after each CL round completes. | | `eval/test_hist_acc` | Accuracy on the *historical* regime, same cadence. Absent when the harness supplies no historical loaders. | +| `eval/test_pre_cl_acc` | Accuracy on the current regime measured *before* the CL round ran, same cadence. | +| `eval/fwt` | Forward transfer: what adapting to this window gained on it. See [Transfer Metrics](#transfer-metrics). | +| `eval/bwt` | Backward transfer: how much past tasks moved since they were learned. Absent on the first drift event. | **`drift/`** — the detector, logged once per `detection_interval`. @@ -101,6 +104,61 @@ The whole `Config` is recorded as run config/params, so `update_mode`, `detector_name`, and the detector hyperparameters are all filterable and groupable in the runs table. +## Transfer Metrics + +`eval/fwt` and `eval/bwt` are indexed by **tasks**, where a task is one drift + event — the window the detector fired on and the CL loop adapted to — so `T` + counts adaptations, not stream windows. + +They are entries of the train-test matrix `R`, where `R[i][j]` is the score on +task `j` after the model finished learning task `i`: + +| Metric | Definition | Reads | +|---|---|---| +| `eval/fwt` | `R[i][i] - R[i-1][i]` | The current window scored after adapting, minus its score before. The gain CL delivered on the task that triggered it. Logged at every drift event, the first included. | +| `eval/bwt` | `(1/(T-1)) * sum over i 0 | adapting raised accuracy on the window — CL helped | adapting raised error — **CL hurt** | +| `fwt` < 0 | CL hurt | **CL helped** | +| `bwt` > 0 | past tasks score better than when learned — backward transfer | past-task error grew — **forgetting** | +| `bwt` < 0 | **forgetting** | past-task error shrank — backward transfer | + +Check the harness's `higher_is_better` before comparing runs across examples. + +One trap on the accuracy side: positive `bwt` means past tasks improved relative +to their own diagonal, which can happen because real backward transfer occurred +*or* because `R[i][i]` was weak to begin with. A short `train.max_iter` leaves the +diagonal undertrained and will manufacture positive `bwt` that means nothing. +Sanity-check `eval/test_curr_acc` at each event before reading a positive value as +transfer. + +Two limits worth knowing. `BaseModelHarness.max_task_records` (default 50) caps +the registry; past that, `bwt` averages over the retained tasks rather than all +`T-1`. And `eval/bwt` differs from `eval/test_hist_acc` on purpose — +`test_hist_acc` is a single *pooled* evaluation over the concatenated history and +carries no reference point, so it conflates forgetting with windows that were +simply harder. + +This `fwt` is the CL gain on the triggering task. It is not the Lopez-Paz & +Ranzato form `R[i-1][i] - b_i`, which measures zero-shot transfer against a +baseline model `b_i`; no baseline term is computed here. + ## Reading the Charts Both figures below are the W&B workspace view on the MNIST example: runs logged @@ -159,6 +217,9 @@ The green-versus-blue contrast is the argument for replay in one image, and it i the contrast the live-stream figure cannot show. It is also what `eval/test_hist_acc` measures continuously during a normal run — see `mix_historic_data` in [`continuous_learning.md`](continuous_learning.md). +`eval/bwt` puts a number on the same effect per drift event, against each task's +own starting point rather than a pooled average — see +[Transfer Metrics](#transfer-metrics). ### Cross-Referencing Other Stages diff --git a/src/apeiron/model/torch_model_harness.py b/src/apeiron/model/torch_model_harness.py index bb7ee9d..07c2c29 100644 --- a/src/apeiron/model/torch_model_harness.py +++ b/src/apeiron/model/torch_model_harness.py @@ -5,7 +5,7 @@ import torch from torch import nn, Tensor -from torch.utils.data import DataLoader +from torch.utils.data import DataLoader, TensorDataset from torch.optim import Optimizer from apeiron.config.configuration import Config @@ -34,6 +34,11 @@ def __init__(self, cfg: Config, model: nn.Module): self.eval_metrics: Dict[str, MetricFn] = {} + # One entry per drift event, oldest first: the frozen validation split of + # the window that was adapted to, paired with R[i][i] (see register_task). + self._task_records: List[Tuple[DataLoader, List[float]]] = [] + self.max_task_records: int = 50 + @abstractmethod def get_optmizer(self) -> Optimizer: """ @@ -104,13 +109,13 @@ def _to_scalar(x: Tensor | float) -> float: return float(x) @torch.no_grad() - def eval(self) -> List[float]: + def _eval_loader(self, loader: DataLoader) -> List[float]: """Stream over batches; return mean(metric) over batches (order preserved).""" self.model.eval() sums = [0.0 for _ in self.eval_metrics] counts = [0 for _ in self.eval_metrics] - for batch in self.get_train_dataloaders()[1]: # assumes iterable + for batch in loader: # assumes iterable x, y = self._unpack(batch) x, y = x.to(self.cfg.device), y.to(self.cfg.device) @@ -140,6 +145,11 @@ def eval(self) -> List[float]: return [s / c for s, c in zip(sums, counts)] + @torch.no_grad() + def eval(self) -> List[float]: + """Stream over batches; return mean(metric) over batches (order preserved).""" + return self._eval_loader(self.get_train_dataloaders()[1]) + @torch.no_grad() def history_eval(self) -> Optional[List[float]]: """Stream over batches; return mean(metric) over batches (order preserved). @@ -150,39 +160,63 @@ def history_eval(self) -> Optional[List[float]]: if hist_loaders is None or hist_loaders[1] is None: return None - self.model.eval() - sums = [0.0 for _ in self.eval_metrics] - counts = [0 for _ in self.eval_metrics] + return self._eval_loader(hist_loaders[1]) + + # ----- per-task evaluation (train-test matrix R) ----- + + def register_task(self, diagonal_metrics: List[float]) -> None: + """Record the task just finished so later events can measure forgetting. - for batch in hist_loaders[1]: + A *task* is one drift event: the window the detector fired on and the CL + loop adapted to. This freezes that window's validation split into a + standalone eval set and stores it alongside ``diagonal_metrics`` -- + ``R[i][i]``, the score on the window measured right after adapting to it. + Both travel in the same record so eviction can never misalign a task's + eval set from its diagonal. + + The split is copied into memory rather than referenced: the harness drops + its window tensors as the stream advances, and a plain ``DataLoader`` + reference would keep the whole window alive through a view. + + :param diagonal_metrics: ``eval()`` output for the current window, taken + after the CL loop finished. + :type diagonal_metrics: List[float] + """ + xs: List[Tensor] = [] + ys: List[Tensor] = [] + for batch in self.get_train_dataloaders()[1]: x, y = self._unpack(batch) - x, y = x.to(self.cfg.device), y.to(self.cfg.device) + xs.append(x.detach().cpu().clone()) + ys.append(y.detach().cpu().clone()) - # TODO: Add cuda amp support later. Needs config entry for amp - # if self.cfg.amp: + frozen = DataLoader( + TensorDataset(torch.cat(xs), torch.cat(ys)), + batch_size=self.cfg.train.batch_size, + shuffle=False, + ) + self._task_records.append((frozen, list(diagonal_metrics))) - # with torch.autocast( - # device_type=self.device.type, - # dtype=( - # torch.float16 if self.device.type == "cuda" else torch.bfloat16 - # ), - # ): - # y_hat = self.model(x) - # else: - y_hat = self.model(x) + # Cap retained tasks; BWT then averages over the surviving ones. + while len(self._task_records) > self.max_task_records: + self._task_records.pop(0) - batch_size = y.shape[0] - for i, m in enumerate(self.eval_metrics.values()): - metric_value = self._to_scalar(m(y_hat, y)) - # For metrics that return percentages (like accuracy), we need to - # convert back to counts for proper averaging across variable batch sizes - sums[i] += metric_value * batch_size - counts[i] += batch_size + @torch.no_grad() + def eval_past_tasks(self) -> List[List[float]]: + """Score the current model on every registered task's frozen eval set. - if counts[0] == 0: - raise RuntimeError("Empty loader: nothing to evaluate.") + Returns row ``T`` of the train-test matrix below the diagonal -- + ``[R[T][i] for i < T]``, oldest task first, index-aligned with + :attr:`task_diagonals`. Empty until at least one task is registered. + """ + return [self._eval_loader(loader) for loader, _ in self._task_records] - return [s / c for s, c in zip(sums, counts)] + @property + def task_diagonals(self) -> List[List[float]]: + """``R[i][i]`` per registered task, oldest first. + + Index-aligned with :meth:`eval_past_tasks`. + """ + return [diagonal for _, diagonal in self._task_records] @property def ckpts_enabled(self) -> bool: diff --git a/src/apeiron/training/continuous_trainer.py b/src/apeiron/training/continuous_trainer.py index 8ac1f50..6e5925e 100644 --- a/src/apeiron/training/continuous_trainer.py +++ b/src/apeiron/training/continuous_trainer.py @@ -83,6 +83,41 @@ def _log_validation( # increment=False: annotate the CL round's step, do not advance it. logger.log(payload, commit=False, increment=False) + def compute_bwt(self, metric_index: int = 0) -> Optional[float]: + """Backward transfer for the task that just finished adapting. + + ``BWT = (1/(T-1)) * sum_{i