GRACE paper claims 8x fewer tokens, 11.1x lower latency for video diffusion

GRACE, a new paper from Jiyoung Kim, says it can shrink the number of tokens a video diffusion model has to process while matching the original pipeline's generation quality on VBench in the reported test. The paper, "GRACE: Generation-aware latent compression for efficient video generation," was posted to arXiv on Wednesday, October 7 (43 pages, 24 figures, filed under Computer Vision).

The headline numbers: applied to Wan2.1-I2V-14B at 480x832x81, GRACE reduces token count by 8x and latency by 11.1x, "while matching the generation quality of the pretrained pipeline before compression on VBench," according to the abstract.

The context: compressing a video autoencoder harder normally degrades reconstruction, and recovering it takes more channels, which slows DiT convergence. Pretrained DiTs also tend to need retraining or costly adaptation. GRACE is a two-stage framework. It keeps a frozen base latent, learns a residual latent for what compression loses, and aligns the compressed latent with the pretrained one in the frozen DiT's feature space. It then adapts the DiT with lightweight fine-tuning and asymmetric denoising, base ahead of residual.

My take: the headline result is for Wan2.1-I2V-14B at 480x832x81, with quality assessed on VBench. The abstract does not say what hardware produced the 11.1x, and it does not list co-authors. VBench parity is a claim about VBench. Read the 43 pages before the speedup gets a parade.