Skip to content

Add forward and backward transfer metrics - #119

Merged
anagainaru merged 2 commits into
mainfrom
metrics
Aug 13, 2026
Merged

Add forward and backward transfer metrics#119
anagainaru merged 2 commits into
mainfrom
metrics

Conversation

@anagainaru

Copy link
Copy Markdown
Collaborator

Adds eval/fwt and eval/bwt to the metrics logged for a run, so a run can answer whether continual learning is actually helping the window it fired on, and whether it's paying for that by forgetting earlier ones.

Without this PR a run reports eval/test_curr_acc and eval/test_hist_acc. test_hist_acc is a single pooled score over concatenated history with no reference point, so it conflates forgetting with windows that were simply harder. The transfer metrics fix that by scoring each past task against its own starting point.

What's logged

A task is one drift event — the window the detector fired on and the CL loop adapted to — so T counts adaptations, not stream windows. Both metrics are entries of the train-test matrix R, where R[i][j] is the score on task j after finishing task i:

Metric Definition Reads
eval/fwt R[i][i] - R[i-1][i] The gain adapting delivered on the triggering window. Logged at every drift event, first included.
eval/bwt (1/(T-1)) · Σ_{i<T} (R[T][i] - R[i][i]) Each past task scored now, minus its score right after it was learned. Absent on the first drift event, where the sum is empty.
eval/test_pre_cl_acc R[i-1][i] The pre-CL score, logged directly so fwt is reconstructible from the CSV.

Sign convention

Both metrics are raw differences of R, so the sign inherits the metric's direction. With accuracy (higher better) negative bwt means forgetting; with a lower-is-better metric such as SLAC-FEL's MAE, positive bwt means forgetting. Check the harness's higher_is_better before comparing across examples.

One trap worth noting: positive bwt on accuracy can mean real backward transfer or a weak R[i][i] to begin with — a short train.max_iter leaves the diagonal undertrained and manufactures meaningless positive values. Sanity-check eval/test_curr_acc per event first.

This fwt is the CL gain on the triggering task, not the Lopez-Paz & Ranzato form R[i-1][i] - b_i; no baseline model term is computed.

Example output

For MNIST:

Screenshot 2026-08-13 at 4 36 19 PM

@anagainaru
anagainaru requested a review from krm9c August 13, 2026 20:36
@anagainaru anagainaru added the Deployment Issues and PRs related to the deployment of the model back in the system label Aug 13, 2026
@krm9c

krm9c commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Okay, the metric calculation looks correct and the mnist plots look correct.

@krm9c krm9c left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, the plots and the calculation looks correct to me.

@anagainaru
anagainaru merged commit cf0b5c5 into main Aug 13, 2026
3 checks passed
@anagainaru
anagainaru deleted the metrics branch August 13, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Deployment Issues and PRs related to the deployment of the model back in the system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants