CogView3 & CogView-3Plus

Read this in Chinese

Experience the CogView3-Plus-3B model online on 🤗 Huggingface Space

📚 Check out the paper

👋 Join our WeChat

📍 Visit Qingyan and API Platform for larger-scale commercial video generation models.

Project Updates

• 🔥🔥 2024/10/13: We have adapted and open-sourced the CogView-3Plus-3B model in the diffusers version. You can experience it online.
• 🔥 2024/9/29: We have open-sourced CogView3 and CogView-3Plus-3B. CogView3 is a text-to-image system based on cascaded diffusion, utilizing a relay diffusion framework. CogView-3Plus is a series of newly developed text-to-image models based on Diffusion Transformers.

Model Introduction

CogView-3-Plus builds upon CogView3 (ECCV'24) by introducing the latest DiT framework for further overall performance improvements. CogView-3-Plus uses the Zero-SNR diffusion noise scheduling and incorporates a joint text-image attention mechanism. Compared to the commonly used MMDiT structure, it effectively reduces training and inference costs while maintaining the model's basic capabilities. CogView-3Plus utilizes a VAE with a latent dimension of 16.

The table below shows the list of text-to-image models we currently offer along with their basic information. At present, all models are only available in the SAT version, but we are participating in the development of the diffusers version.

Model Name	CogView3-Base-3B	CogView3-Base-3B-distill	CogView3-Plus-3B
Model Description	The base and relay stage models of CogView3, supporting 512x512 text-to-image generation and 2x super-resolution generation.	The distilled version of CogView3, with 4 and 1 step sampling in two stages (or 8 and 2 steps).	The DiT version image generation model, supporting image generation ranging from 512 to 2048.
Resolution	512 * 512		512 <= H, W <= 2048 H * W <= 2^{21} H, W \mod 32 = 0
Inference Precision	FP16 (recommended), BF16, FP32		BF16* (recommended), FP16, FP32
Memory Usage (bs = 4)	17G	64G	30G (2048 * 2048) 20G (1024 * 1024)
Prompt Language	English*
Maximum Prompt Length	225 Tokens		224 Tokens
Download Link (SAT)	SAT
Download Link (Diffusers)	Not Adapted		🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel

Data Explanation

• All inference tests were conducted on a single A100 GPU with a batch size of 4, using PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to save memory.
• The models only support English input. Other languages can be translated into English when refining with large models.
• This test environment uses the SAT framework. Many optimization points are not yet complete, and we will work with the community to create a version of the model for the diffusers library. Once the diffusers repository is supported, we will test using diffusers. The release is expected in November 2024.

Quick Start

Prompt Optimization

Although CogView3 series models are trained with long image descriptions, we highly recommend rewriting prompts using large language models (LLMs) before generating text-to-image, as this will significantly improve generation quality.

We provide an example script. We suggest running this script to refine the prompt:

python prompt_optimize.py --api_key "Zhipu AI API Key" --prompt {your prompt} --base_url "https://open.bigmodel.cn/api/paas/v4" --model "glm-4-plus"

Inference Model (Diffusers)

First, ensure the diffusers library is installed from source.

pip install git+https://github.com/huggingface/diffusers.git

Then, run the following code:

from diffusers import CogView3PlusPipeline
import torch

pipe = CogView3PlusPipeline.from_pretrained("THUDM/CogView3-Plus-3B", torch_dtype=torch.float16).to("cuda")

# Enable it to reduce GPU memory usage
pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

prompt = "A vibrant cherry red sports car sits proudly under the gleaming sun, its polished exterior smooth and flawless, casting a mirror-like reflection. The car features a low, aerodynamic body, angular headlights that gaze forward like predatory eyes, and a set of black, high-gloss racing rims that contrast starkly with the red. A subtle hint of chrome embellishes the grille and exhaust, while the tinted windows suggest a luxurious and private interior. The scene conveys a sense of speed and elegance, the car appearing as if it's about to burst into a sprint along a coastal road, with the ocean's azure waves crashing in the background."
image = pipe(
    prompt=prompt,
    guidance_scale=7.0,
    num_images_per_prompt=1,
    num_inference_steps=50,
    width=1024,
    height=1024,
).images[0]

image.save("cogview3.png")

For more inference code, please refer to inference. This folder also contains a simple WEBUI code wrapped with Gradio.

Inference Model (SAT)

Please check the sat tutorial for step-by-step instructions on model inference.

Open Source Plan

Since the project is in its early stages, we are working on the following:

• Fine-tuning the SAT version of CogView3-Plus-3B, including SFT and LoRA fine-tuning
• Inference with the Diffusers library version of the CogView3-Plus-3B model
• Fine-tuning the Diffusers library version of the CogView3-Plus-3B model
• Related work for the CogView3-Plus-3B model, including ControlNet and other tasks.

CogView3 (ECCV'24)

Official paper repository: CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion

CogView3 is a novel text-to-image generation system using relay diffusion. It breaks down the process of generating high-resolution images into multiple stages. Through the relay super-resolution process, Gaussian noise is added to low-resolution generation results, and the diffusion process begins from these noisy images. Our results show that CogView3 outperforms SDXL with a winning rate of 77.0%. Additionally, through progressive distillation of the diffusion model, CogView3 can generate comparable results while reducing inference time to only 1/10th of SDXL's.

Comparison results from human evaluations:

Citation

🌟 If you find our work helpful, feel free to cite our paper and leave a star.

@article{zheng2024cogview3,
  title={Cogview3: Finer and faster text-to-image generation via relay diffusion},
  author={Zheng, Wendi and Teng, Jiayan and Yang, Zhuoyi and Wang, Weihan and Chen, Jidong and Gu, Xiaotao and Dong, Yuxiao and Ding, Ming and Tang, Jie},
  journal={arXiv preprint arXiv:2403.05121},
  year={2024}
}

We welcome your contributions! Click here for more information.

Model License

This codebase is released under the Apache 2.0 License.

The CogView3-Base, CogView3-Relay, and CogView3-Plus models (including the UNet module, Transformers module, and VAE module) are released under the Apache 2.0 License.