On-Policy Self-Distillation: A New Approach to Continual Learning

On-policy self-distillation (OPSD)¹ has taken the world of machine learning research by storm. Although OPSD frameworks rely heavily on older, more established approaches, the first papers on the topic were only submitted in late January 2026, just under nine months ago at the time of writing this article [1, 2]. Several dozen papers have followed within that brief timespan [3], covering everything from extensions [3] to limitations [4, 5].

In this article, we’ll explore the literature on OPSD, mainly through the lens of the Self-Distillation Fine-Tuning (SDFT) framework introduced by Shenfeld et al. in Self-distillation enables continual learning [1]. As we’ll see, the term “continual learning” in the paper’s title refers to a model’s ability to effectively and dynamically learn from a datastream. We’ll start by examining the problem OPSD frameworks set out to solve and the approaches they build on.

After that, we’ll look at how SDFT works and the theoretical explanations for its success. These foundations motivate the experiments described in the paper, with their promising results lending credence to the prospect of OPSD being widely applicable. Finally, we’ll touch on two central concerns that have surfaced regarding OPSD.

Catastrophic Forgetting: Why Continual Learning is Hard

As ever more powerful foundation models have hit the market, the sheer number of tasks they can perform well out of the box has skyrocketed. These models are pre-trained on enormous amounts of data using an incredible amount of computational resources. Pre-training is probably the first thing that comes to mind for most people when talking about “training” machine learning models in general. It is the stage during which a model attains its fundamental capabilities and knowledge.

Post-training complements pre-training by teaching a model to channel what it has learned during pre-training for a certain purpose, to behave in desirable ways, or to learn new skills. Post-training goals could be anything from better instruction following to improved performance on difficult math problems. Foundation models also undergo extensive post-training before being released for use.

For many highly complex or customized tasks, however, further training remains necessary. Pre-training a custom model from scratch would be prohibitively expensive and impractical. Instead, we carry out additional post-training on foundation models. Fine-tuning is a particular form of post-training. Conducting it a single time could work if we want to specialize the model for a particular domain. But what if we want to build on a vision-language-action (VLA) model to enable a robot to carry out a variety of intricate tasks? Or to post-train an LLM for better cross-domain performance in a low- to mid-resource language²?

The key problem that arises in such cases is catastrophic forgetting: models forgetting how to handle older tasks as a result of learning new ones. When catastrophic forgetting happens during post-training, it often happens rapidly. Naturally, as we train a model on the new tasks we want to teach it, we influence its weights. This is the mechanism by which learning happens. We gradually “pull” the weights away from those of the base model.

However, if we throw too much at the model — be it too much new data or even a small amount of new data that is too radically out-of-distribution compared to what it has seen before — we risk pulling so hard that we interfere with its ability to carry out the tasks it was previously trained to conduct. What constitutes “too much” in this case is often much less than what you would expect from a human.

Continual learning, as the name implies, is the ability of a model to learn continually from a potentially endless and highly variable stream of data while avoiding catastrophic forgetting. Continual learning, done properly, yields models that are capable of acquiring an arbitrary number of new skills without meaningfully sacrificing prior abilities.

Figure 1: Search interest for “Continual Learning”, January 2025 – July 2026. Image by the author. Source: Google Trends.

Interest in continual learning has spiked over the past year (see Figure 1). Real headway has been made in that time. The unquestionable improvement in available open-weight foundation models, paired with new post-training methods, has ushered in an era where true continual learning seems more applicable to a wider range of real-world problems than ever.

A key family of methods introduced in that time, as mentioned above, is OPSD, with a particular member of that family being SDFT. Having understood the problem these approaches aim to solve, we can review the more traditional approaches SDFT builds upon and the theoretical and empirical results thanks to which OPSD has drawn so much attention since the beginning of 2026.

Distill Thyself

Knowledge distillation is a well-established machine learning technique. It has garnered significant attention over the past year, much of it negative and centering on accusations of distillation attacks [6]. However, there’s nothing inherently bad about distillation. It is a highly effective and widely used technique. The idea at its core is simple: use a bigger, more complex model as a teacher to train a student model to replicate its outputs.

Self-distillation puts a twist on the classic approach. Rather than using a different, more powerful model in the role of the teacher, both the teacher and the student are copies of the exact same model. The only things that distinguish the teacher from the student are that the teacher is given access to privileged information and that its parameters are updated slightly differently. As we’ll see below, this can be beneficial in reducing catastrophic forgetting.

OPSD methods in general, and SDFT in particular, employ self-distillation. (That’s what the “SD” in both of them stands for.) In SDFT, the teacher is conditioned on expert demonstrations for each query (see Figure 2). Theoretically, these expert demonstrations could include anything that might give the teacher an edge. They could be brief justifications of correct answers, instructions on how to reach such correct answers, or even entire reasoning traces where thinking models are concerned.

In SDFT in particular, for multiple choice questions drawn from a chemistry subset of the SciKnowEval benchmark [7], the teacher model is given the correct answer and a brief explanation on how to derive it (e.g., the process needed to determine the molar mass of some particular molecule). The student and teacher are both instantiations of (non-thinking) Qwen-2.5-7B-Instruct. The chemistry expert demonstrations are generated by the much stronger GPT-4o [1].

Figure 2: Diagram describing the architecture of SDFT. Each query is paired with a corresponding expert demonstration, e.g., an explanation of a correct answer to the query. Only the teacher sees the demonstrations. The student and the teacher are initialized as the same model with identical parameters. Teacher parameters are updated as an exponential moving average of student parameters. The loss term is the reverse KL divergence between student and teacher next-token distributions. These distributions differ due to the privileged information asymmetry between the two. Image by I. Shenfeld, M. Damani, J. Hübotter, P. Agrawal from Self-distillation enables continual learning (use under CC-BY 4.0 license).

For each token generated by the student while answering a query, the student and the teacher each produce a next-token distribution, denoted P and Q respectively in Figure 2. The difference between P and Q is that Q is informed by the privileged demonstrations to which only the teacher has access. At each step, the training objective is to minimize the reverse³ KL divergence between P and Q. Computing the divergence is the only thing Q is used for, as the teacher never actually generates any tokens.

Training directly updates student parameters at each step. Teacher parameters are initially identical to student parameters. There are three ways teacher parameters can be updated throughout training.

• No updates: The teacher weights are frozen in their initial state. Shenfeld et al. [1] report that frozen teacher weights prevent the teacher from improving alongside the student during training, leading to suboptimal performance.

• Student-identical updates: That is, directly using the continuously updated student model as the teacher throughout training. According to Shenfeld et al. [1], this lowers the stability of training. An overly similar student-teacher pair develops an echo chamber in which variations can send training off the rails.

• Averaged updates: Teacher parameters are updated as an exponential moving average (EMA) of student parameters. This is the approach adopted in SDFT, as it strikes the right balance between the two extremes above — keeping the teacher meaningfully distinct from the student while still allowing it to improve as training progresses.

The EMA update mechanism ensures that the teacher’s parameters evolve smoothly over time, incorporating new information from the student without collapsing into an identical copy. This dynamic balance is central to SDFT’s effectiveness at enabling continual learning while resisting catastrophic forgetting.