Gradient Episodic Memory
GEM learns new tasks while trying not to damage performance on earlier ones. It stores a small episodic memory and treats gradients computed from those remembered examples as constraints.
Continual learning
Say a classifier learns to recognize one pair of digit classes on Monday and another on Tuesday. Tuesday’s training loss can fall while Monday’s accuracy falls too. At that point the optimizer is doing exactly what its training batch asks of it. The problem is that Monday’s task is missing from the calculation.
Gradient Episodic Memory, or GEM, keeps a small batch from each earlier task. Before applying an update, it asks what that update would do to the loss on those batches. If the usual update would raise one of those losses to first order, GEM looks for the smallest change to the gradient that avoids the conflict.
How two gradients can disagree
Let g be the gradient of the current loss, and m be the gradient from one remembered batch. Gradient descent takes a small step in the direction −g. The old loss changes by approximately −η(m · g), where η is the learning rate. If m · g is positive, that estimate says the old loss goes down. If it is negative, the old loss goes up.
The minus sign matters. The gradients point toward increasing loss, even though the optimizer moves in the opposite direction. In this example there are only two parameters, so you can check everything with two numbers.
Current gradient g: (2, −1)
Remembered gradient m: (0, 1)
Dot product m · g: −1
First-order change: +η, an increase in old lossKeep the part of the update that still works
Here the positive first coordinate in g causes no trouble. The conflict comes entirely from its second coordinate. GEM projects (2, −1) onto the allowed region, which in this case is any gradient whose second coordinate is at least zero. The closest allowed point is (2, 0). The optimizer can still move along the first coordinate and the first-order change in the remembered loss becomes zero.
In a model with several earlier tasks, every remembered batch supplies its own constraint. The projection has to satisfy them all at once, because fixing one conflict can bring back a different one. GEM solves a quadratic program for that projection. The paper derives a smaller version whose number of optimization variables depends on the number of earlier tasks rather than the model’s parameter count.
What the memory can tell you
An old batch is a sample, so a safe update on that batch can still hurt examples the model has never seen. The test also uses a local approximation to the loss, which can become inaccurate for a large step. There is a cost for each earlier task too. You have to store some examples and run a backward pass to get its gradient.
A useful way to read the paper’s experiments is to ask how an earlier task scored when it was first learned and how it scored after the last task. The paper uses that difference to measure backward transfer on task streams built from MNIST and CIFAR-100. A single average at the end can hide the cases where the model improved on recent tasks at the expense of early ones.