Self-distillation has become one of those terms in machine learning that sounds more philosophical than technical. Exploring how neural networks can teach themselves to become better versions of themselves.