1

MAPL treats inter-stage activation compression in pipeline parallelism as a learnable orthogonal projection under Stiefel manifold constraints, letting each stage adapt its own task-optimal subspace. Combined with factorized anchor embeddings and residual vector quantization, it achieves high compression with negligible performance loss across LLaMA models from 150M to 1B parameters.

A PyTorch library that makes learned optimizers accessible to the broader ML community with CUDA-accelerated implementations with substantial speedups (5x improvement for ViT training) and integrates seamlessly with existing PyTorch workflows, enabling practical application of learned optimization to real-world large-scale tasks.

We identify the uncontrolled growth of weight-update spectral norms as the key instability in training natively low-rank LLMs, and introduce Spectron, a lightweight spectral renormalization technique that enables stable end-to-end factorized pretraining with compute-optimal scaling laws and improved inference efficiency.

We demonstrate that infinite learning rate schedules consistently outperform widely-used repeated cosine decay for continual pre-training under distribution shifts across both vision and language models, providing a more effective alternative for large-scale self-supervised learning without catastrophic forgetting.

This study investigates the capabilities of video diffusion models in generating human motion from text prompts, revealing their strengths in common motions and limitations in rare or complex movements.

Learning novel concepts, remembering previous knowledge, and adapting it to future tasks occur simultaneously throughout a human’s …

Adding knowledge to the model without destroying its generalization by finetuning small set of parameters

Continual zero-shot learning involves learning seen classes incrementally while improving the ability to recognize unseen or …

A baseline that performs better without training in continual learning benchmarks