Add forward and backward transfer metrics - #119
Merged
Merged
Conversation
Collaborator
|
Okay, the metric calculation looks correct and the mnist plots look correct. |
krm9c
approved these changes
Aug 13, 2026
krm9c
left a comment
Collaborator
There was a problem hiding this comment.
I think, the plots and the calculation looks correct to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
eval/fwtandeval/bwtto the metrics logged for a run, so a run can answer whether continual learning is actually helping the window it fired on, and whether it's paying for that by forgetting earlier ones.Without this PR a run reports
eval/test_curr_accandeval/test_hist_acc.test_hist_accis a single pooled score over concatenated history with no reference point, so it conflates forgetting with windows that were simply harder. The transfer metrics fix that by scoring each past task against its own starting point.What's logged
A task is one drift event — the window the detector fired on and the CL loop adapted to — so
Tcounts adaptations, not stream windows. Both metrics are entries of the train-test matrixR, whereR[i][j]is the score on taskjafter finishing taski:eval/fwtR[i][i] - R[i-1][i]eval/bwt(1/(T-1)) · Σ_{i<T} (R[T][i] - R[i][i])eval/test_pre_cl_accR[i-1][i]fwtis reconstructible from the CSV.Sign convention
Both metrics are raw differences of
R, so the sign inherits the metric's direction. With accuracy (higher better) negativebwtmeans forgetting; with a lower-is-better metric such as SLAC-FEL's MAE, positivebwtmeans forgetting. Check the harness'shigher_is_betterbefore comparing across examples.One trap worth noting: positive
bwton accuracy can mean real backward transfer or a weakR[i][i]to begin with — a shorttrain.max_iterleaves the diagonal undertrained and manufactures meaningless positive values. Sanity-checkeval/test_curr_accper event first.This
fwtis the CL gain on the triggering task, not the Lopez-Paz & Ranzato formR[i-1][i] - b_i; no baseline model term is computed.Example output
For MNIST: