Jodi: Unification of Visual Generation and Understanding via Joint Modeling

Yifeng Xu1,2, Zhenliang He1, Meina Kan1,2, Shiguang Shan1,2, Xilin Chen1,2
1State Key Lab of AI Safety, Institute of Computing Technology, CAS, China
2University of Chinese Academy of Sciences, China
arXiv Code

🤗

 Model
banner

Jodi can perform (a) joint generation, (b) controllable generation, and (c) image perception in a unified diffusion model.

Abstract

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.


overview

Results

Joint Generation: p(x,y1,y2,...)

Controllable Generation: p(x|y1,y2,...)

Image Perception: p(y1,y2,...|x)

Joint-1.6M Dataset

We collect images with high quality and diversity from several publicly available sources, including Subjects200K, Aesthetic-4K, and Pexels. All of these images have resolutions over 1024×1024, which is advantageous for training a high-resolution generative model. As these datasets lack labels, we use state-of-the-art predictors to automatically annotate the data with labels corresponding to 7 specific domains. Specifically, we employ Informative Drawings to generate line arts, PiDiNet to extract edge maps, Depth Anything V2 and Lotus to estimate depth maps, Lotus to estimate normal maps, RGB2X to estimate albedos, Oneformer to predict segmentation colormaps, and Openpose to predict human skeletons. In this manner, we construct a dataset containing 200,000 images with corresponding 7×200,000 predicted labels. Furthermore, we use BLIP2-OPT-2.7b and Qwen2-VL-7b-Instruct to generate captions. The former tends to provide a concise description of the main subject in the image, while the latter tends to give a long paragraph that describes the subject, background, and the overall atmosphere in detail.

BibTex

@article{xu2025jodi,
  title={Jodi: Unification of Visual Generation and Understanding via Joint Modeling},
  author={Xu, Yifeng and He, Zhenliang and Kan, Meina and Shan, Shiguang and Chen, Xilin},
  journal={arXiv preprint arXiv:2505.19084},
  year={2025}
}