Back to feed
AR
arXiv CS.AI
7/29/2026
RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

Short summary

RoCo-ACE introduces a rollout-conditioned online distillation objective for injecting new knowledge into pretrained multimodal LLMs while minimizing drift in existing behavior. RoCo reallocates distillation weight to reference-supported rollout tokens using likelihood contrast, while ACE adds sparse correction for authoritative facts omitted from rollouts. Across multiple settings and base models, RoCo-ACE achieves best injected-knowledge accuracy while keeping retention close to the base model.

  • •RoCo-ACE is a distillation objective for knowledge injection that minimizes behavioral drift
  • •RoCo reweights reference-supported tokens; ACE corrects omitted authoritative facts
  • •Best injected-knowledge accuracy across multiple settings while retaining base model behavior

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv CS.AI

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations
arXiv cs.CL
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
arXiv cs.CL
Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
arXiv cs.CL
SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
arXiv cs.CL