Pooled CLIP embeddings are useless for conditioning but powerful as guidance signals.
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.
Pooled CLIP embeddings provide minimal contribution to modern diffusion transformers like FLUX and HiDream.
Repurposing pooled embeddings as active guidance signals instead of passive conditioning dramatically improves effectiveness.
Guidance signals created from positive-negative prompt embedding differences steer generation toward desirable aesthetic qualities.
The paper only tests on specific state-of-the-art models, limiting generalizability across all diffusion transformer architectures.
Effectiveness of the guidance approach depends on carefully crafted positive and negative prompts for desired results.
Shenghai Yuan, Yuanyang Yin +4
We introduce Helios, the first 14B video generation model that runs at 19.5 FPS on a single NVIDIA H100 GPU and supports minute-scale generation while matching the quality of a strong baseline. We mak...
Yiying Yang, Wei Cheng +6
OmniLottie is a versatile framework that generates high quality vector animations from multi-modal instructions. For flexible motion and visual content control, we focus on Lottie, a light weight JSON...
Zengbin Wang, Xuecai Hu +4
Text-to-image (T2I) models have achieved remarkable success in generating high-fidelity images, but they often fail in handling complex spatial relationships, e.g., spatial perception, reasoning, or i...
Grigorii Alekseenko, Aleksandr Gordeev +8
Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside high...
Hojung Jung, Rodrigo Hormazabal +6
Molecular generation with diffusion models has emerged as a promising direction for AI-driven drug discovery and materials science. While graph diffusion models have been widely adopted due to the dis...
