OpenAI 生成视频模型 Sora 报告翻译

文章概述

本文介绍了一个名为Sora的模型，它是一种基于扩散变换的视频生成器。与传统的视频生成方法不同，Sora可以在原始视频画质和大小上进行训练，并且可以生成各种宽度和高度的视频。此外，Sora还可以通过提供文本描述或者指令来启动训练，并且可以生成高质量的图像和视频。文章还讨论了Sora在语言理解、图像生成和视频编辑等方面的应用，并提到了它在视频创作和模拟方面的潜力。然而，文章也指出了Sora目前存在的一些限制，如缺乏准确的物理模拟和对象状态的变化等。

文章速读

世界模拟器

这一章节主要介绍了Sora模型的两个方面：一是如何将各种类型的视觉数据转化为统一的表示形式进行大规模训练；二是对Sora模型的能力和限制进行了定性评估。与以往的研究不同的是，Sora模型能够生成各种时长、分辨率和比例的视频和图像，并且采用了类似于语言模型中的“token”的方式来处理视觉数据，即使用了“spacetime patches”。此外，该文还介绍了一种压缩视频的方法以及对应的解码器模型。最后，作者通过对比不同计算资源下的样本质量，证明了随着训练计算量的增加，Sora模型的表现会显著提高。

Sora 全新视频生成模型，支持多种比例和语言理解

这一章节介绍了Sora模型的训练方式和应用场景。该模型可以生成不同分辨率、不同比例的视频，并且能够根据用户输入的图片或文字进行动画制作。此外，通过重新标注视频并使用GPT模型生成详细的文字描述，可以提高视频的质量和准确度。最后，该模型还可以用于编辑现有的图像和视频，实现更加灵活的应用场景。

Sora的无限可能

这一章节介绍了名为Sora的视频生成模型的一些功能和特点。它能够对输入的视频进行风格和环境的零样本转换，并且可以实现无缝的无限循环。此外，Sora还可以通过逐渐插值两个完全不同的视频来创建平滑的过渡效果。除此之外，Sora还具有图像生成的能力，可以通过排列高斯噪声的方式生成大小可变的图像。最后，该文指出，随着训练规模的扩大，Sora还展现出了模拟物理世界、数字世界以及其中人物、动物和环境等有趣的能力。

Sora 视频模型的潜力与局限性

这一章节主要在讨论Sora模拟器的局限性以及未来的发展方向。目前Sora模拟器存在许多问题，比如无法准确模拟玻璃破碎等基本物理交互，食物摄入等交互也无法正确改变对象状态。此外，长期样本中的不连贯性和物体自发出现等问题也是常见的失败模式。然而，作者认为Sora模拟器已经展示出了视频模型继续扩展的能力，这是开发能够模拟物理和数字世界及其内部生物、动物和人类的有能力的模拟器的有希望的道路。

1. Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov. "Unsupervised learning of video representations using lstms." International conference on machine learning. PMLR, 2015.↩︎

2. Chiappa, Silvia, et al. "Recurrent environment simulators." arXiv preprint arXiv:1704.02254 (2017).↩︎

3. Ha, David, and Jürgen Schmidhuber. "World models." arXiv preprint arXiv:1803.10122 (2018).↩︎

4. Vondrick, Carl, Hamed Pirsiavash, and Antonio Torralba. "Generating videos with scene dynamics." Advances in neural information processing systems 29 (2016).↩︎

5. Tulyakov, Sergey, et al. "Mocogan: Decomposing motion and content for video generation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.↩︎

6. Clark, Aidan, Jeff Donahue, and Karen Simonyan. "Adversarial video generation on complex datasets." arXiv preprint arXiv:1907.06571 (2019).↩︎

7. Brooks, Tim, et al. "Generating long videos of dynamic scenes." Advances in Neural Information Processing Systems 35 (2022): 31769-31781.↩︎

8. Yan, Wilson, et al. "Videogpt: Video generation using vq-vae and transformers." arXiv preprint arXiv:2104.10157 (2021).↩︎

9. Wu, Chenfei, et al. "Nüwa: Visual synthesis pre-training for neural visual world creation." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.↩︎

10. Ho, Jonathan, et al. "Imagen video: High definition video generation with diffusion models." arXiv preprint arXiv:2210.02303 (2022).↩︎

11. Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.↩︎

12. Gupta, Agrim, et al. "Photorealistic video generation with diffusion models." arXiv preprint arXiv:2312.06662 (2023).↩︎

13. Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).↩︎↩︎

14. Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.↩︎↩︎

15. Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).↩︎↩︎

16. Arnab, Anurag, et al. "Vivit: A video vision transformer." Proceedings of the IEEE/CVF international conference on computer vision. 2021.↩︎↩︎

17. He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.↩︎↩︎

18. Dehghani, Mostafa, et al. "Patch n'Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution." arXiv preprint arXiv:2307.06304 (2023).↩︎↩︎

19. Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.↩︎

20. Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).↩︎

21. Sohl-Dickstein, Jascha, et al. "Deep unsupervised learning using nonequilibrium thermodynamics." International conference on machine learning. PMLR, 2015.↩︎

22. Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.↩︎

23. Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." International Conference on Machine Learning. PMLR, 2021.↩︎

24. Dhariwal, Prafulla, and Alexander Quinn Nichol. "Diffusion Models Beat GANs on Image Synthesis." Advances in Neural Information Processing Systems. 2021.↩︎

25. Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in Neural Information Processing Systems 35 (2022): 26565-26577.↩︎

26. Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.↩︎

27. Chen, Mark, et al. "Generative pretraining from pixels." International conference on machine learning. PMLR, 2020.↩︎

28. Ramesh, Aditya, et al. "Zero-shot text-to-image generation." International Conference on Machine Learning. PMLR, 2021.↩︎

29. Yu, Jiahui, et al. "Scaling autoregressive models for content-rich text-to-image generation." arXiv preprint arXiv:2206.10789 2.3 (2022): 5.↩︎

30. Betker, James, et al. "Improving image generation with better captions." Computer Science. https://cdn.openai.com/papers/dall-e-3. pdf 2.3 (2023): 8↩︎↩︎

31. Ramesh, Aditya, et al. "Hierarchical text-conditional image generation with clip latents." arXiv preprint arXiv:2204.06125 1.2 (2022): 3.↩︎

32. Meng, Chenlin, et al. "Sdedit: Guided image synthesis and editing with stochastic differential equations." arXiv preprint arXiv:2108.01073 (2021).↩︎