During continual learning, the model is trained sequentially on each task. After learning ( \mathcalT t ), the model should perform well on all seen tasks ( \mathcalT 1:t ) without access to previous data. We allow a small episodic memory ( M ) (size ( K )) that stores generated seeds , not real examples.
[5] Zhang, Y., et al. (2024). VLM-CL: A benchmark for continual learning in vision-language models. NeurIPS Datasets Track.
[6] von Oswald, J., et al. (2020). Continual learning with hypernetworks. ICLR.
: Auto-Seed VL2 outperforms all baselines, including ER-VLM with 10× more memory, and beats generative replay by over 13 points on average. The BLEU-4 score on C→F is particularly striking, indicating that generated seeds capture caption semantics well. 6.2 Ablation Study Removing components from Auto-Seed VL2 on C→R: auto seed vl2
[4] Thengane, V., et al. (2023). Continual-CLIP: Fine-tuning CLIP for continual learning. CVPR Workshop.
[7] Khattak, M. U., et al. (2023). MaPLe: Multi-modal prompt learning. CVPR.
This paper is written in a standard academic format (abstract, introduction, methodology, experiments, results, conclusion) and assumes a novel contribution to the fields of continual learning and vision-language models. Author Names Redacted for Blind Review Affiliation Redacted Abstract Vision-Language Models (VLMs) have demonstrated remarkable zero-shot capabilities but suffer from catastrophic forgetting when sequentially fine-tuned on downstream tasks. Traditional continual learning (CL) methods rely on either exemplar replay (which raises privacy concerns) or static prompt pools (which lack adaptability to novel task distributions). We introduce Auto-Seed VL2 , a novel framework for autonomous seed generation that dynamically synthesizes "seed" embeddings—compact, task-representative vectors—without storing real data. Auto-Seed VL2 employs a lightweight meta-generator conditioned on task-specific gradients and a contrastive consistency mechanism to align generated seeds with both visual and textual manifolds. Extensive experiments on four challenging VLM continual learning benchmarks (CIFAR-100 to ImageNet-R, COCO Captions to Flickr30k) show that Auto-Seed VL2 outperforms state-of-the-art methods by 8.7% in average accuracy while reducing memory overhead by 95% compared to exemplar replay. Our analysis further reveals that auto-generated seeds capture inter-task transferable features, enabling forward transfer without explicit rehearsal. 1. Introduction Large-scale pre-trained Vision-Language Models (e.g., CLIP, ALIGN, Flava) have become foundational backbones for multimodal understanding. However, real-world deployment requires these models to adapt continuously to new tasks—new visual domains, novel object categories, or unseen captioning styles—without forgetting previously learned knowledge. This setting, known as Continual Learning (CL), is particularly challenging for VLMs due to the intertwined nature of their dual encoders. During continual learning, the model is trained sequentially
The consistency loss and gradient-conditioned generation are crucial. Seed pruning is memory-efficient without hurting accuracy. We measure FWT: performance on task ( t ) after training on tasks ( 1..t-1 ). Auto-Seed VL2 achieves positive forward transfer (FWT = +4.1%) on VL-CL, meaning seeds from earlier tasks help learn new tasks. ER-VLM shows near-zero FWT; generative replay shows negative transfer due to noisy synthetic images. 7. Analysis and Discussion What do generated seeds encode? We project seeds into CLIP space and compare to real class means. The cosine similarity is 0.89 ± 0.05, indicating faithful representation. However, seeds are more “regularized” – they have lower variance along task-irrelevant directions.
By generating seeds in embedding space rather than pixel space, we avoid the compounding errors of full image generation. The hypernetwork’s meta-learning objective ensures that seeds are discriminative for the original task and compatible with the continually updated VLM.
Auto-Seed VL2 maintains a set of auto-generated seeds ( \mathcalS ) that grows slowly over tasks. Auto-Seed VL2 operates in three phases per task: (1) Seed replay, (2) Online adaptation, (3) Seed update. 4.1 Overall Architecture [5] Zhang, Y
[2] Shin, H., et al. (2017). Continual learning with deep generative replay. NIPS.
[3] Zhou, K., et al. (2022). Learning to prompt for vision-language models. IJCV.
. A seed is a tuple ( s = (v, w) ), where ( v \in \mathbbR^d ) is a visual prototype and ( w \in \mathbbR^d ) is a textual prototype, such that for any example ( (x, y) ) from a past task, ( |f_I(x) - v| ) and ( |f_T(y) - w| ) are small, and ( \textsim(v, w) ) is high.
| Configuration | Avg Acc | Drop | |----------------------------------------|---------|------| | Full Auto-Seed VL2 | 82.2 | — | | w/o consistency loss (( \mathcalL \textconsist )) | 75.4 | -6.8 | | w/o gradient-conditioned generation (random seeds) | 68.9 | -13.3 | | w/o meta-update of ( G \phi ) | 74.1 | -8.1 | | w/o seed pruning (full memory) | 82.0 | -0.2 (ns) |