DPoser: Diffusion Model as Robust 3D Human Pose Prior

Junzhe Lu1, Jing Lin2, Hongkun Dou1,
Ailing Zeng3, Yue Deng1, Yulun Zhang4, and Haoqian Wang2

1Beihang University, 2Tsinghua University, 3International Digital Economy Academy (IDEA), 4ETH Zürich
arXiv Video Code

Abstract

This work targets to construct a robust human pose prior. However, it remains a persistent challenge due to biomechanical constraints and diverse human movements. Traditional priors like VAEs and NDFs often exhibit shortcomings in realism and generalization, notably with unseen noisy poses. To address these issues, we introduce DPoser, a robust and versatile human pose prior built upon diffusion models. DPoser regards various pose-centric tasks as inverse problems and employs variational diffusion sampling for efficient solving. Accordingly, designed with optimization frameworks, DPoser seamlessly benefits human mesh recovery, pose generation, pose completion, and motion denoising tasks. Furthermore, due to the disparity between the articulated poses and structured images, we propose truncated timestep scheduling to enhance the effectiveness of DPoser. Our approach demonstrates considerable enhancements over common uniform scheduling used in image domains, boasting improvements of 5.4\%, 17.2\%, and 3.8\% across human mesh recovery, pose completion, and motion denoising, respectively. Comprehensive experiments demonstrate the superiority of DPoser over existing state-of-the-art pose priors across multiple tasks.

Demo Video

Methodology Overview

Overview of the DPoser Methodology. Panel (a) presents three tasks: human mesh recovery, pose completion, and motion denoising, with omissions like camera optimization and bending loss for clarity. Panel (b) demonstrates the DPoser regularization process, introducing noise to the current pose and applying a one-step denoiser to achieve a denoised pose. \(L_\text{DPoser}\) is computed between the denoised and current pose. Panel (c) outlines the optimization process from initial to current and fitted poses via loss minimization.

Pose Generation

While DPoser’s outputs are visually diverse and realistic, poses generated from competing methods like GMM and Pose-NDF fall short in naturalism, and VPoser exhibits limited diversity.

Pose Completion

Left leg under occlusion.

Trunk under occlusion.

In pose completion, DPoser uniquely offers multiple plausible solutions for partially observed poses, a scenario where competitors often struggle due to limited generalization.

Human Mesh Recovery

Initialization using mean poses and default camera.

Initialization using the CLIFF model.

Description1

EHF dataset

Description2

COCO dataset

Description3

3DPW dataset

Description4

UBody dataset

More examples of DPoser’s fitting.

In Human Mesh Recovery (HMR), DPoser adeptly estimates poses from single images, achieving more natural poses that better align with the image data.

Motion Denoising

DPoser excels in reconstructing lifelike pose sequences from noisy and incomplete 3D joint data. Its application as a single-frame pose prior extends impressively to sequences, ensuring realistic movements.

Citation

@article{dposer,
  title={DPoser: Diffusion Model as Robust 3D Human Pose Prior},
  author={Lu, Junzhe and Lin, Jing and Dou, Hongkun and Zeng, Ailing and Deng, Yue and Zhang, Yulun and Wang, Haoqian},
  journal={arxiv:2312.05541},
  year={2023}
}

The website template was adapted from HumanTOMATO Project.