Gradient / Activation checkpointing

Key TakeawaysGradient Checkpointing is a memory optimization technique used in deep learning to reduce memory consumption during backpropagation.It introduces checkpoint layers at specific points in the network, storing intermediate activations only at these checkpoints, reducing memory requirements.Gradient Checkpointing enables the training of extremely deep neural networks that would be limited by memory constraints without this technique.It is particularly valuable in real-world appli...