The third Workshop on
Reconstruction of Human-Object Interactions (RHOBIN)

Abstract: Multi-person human mesh recovery (HMR) consists in detecting all individuals in a given input image, and predicting the body shape, pose, and 3D location for each detected person. The dominant approaches to this task rely on neural networks trained to output a single prediction for each detected individual. In contrast, we propose CondiMen, a method that outputs a joint parametric distribution over likely poses, body shapes, intrinsics and distances to the camera, using a Bayesian network. This approach offers several advantages. First, a probability distribution can handle some inherent ambiguities of this task – such as the uncertainty between a person’s size and their distance to the camera, or more generally the loss of information that occurs when projecting 3D data onto a 2D image. Second, the output distribution can be combined with additional information to produce better predictions, by using e.g. known camera or body shape parameters, or by exploiting multi-view observations. Third, one can efficiently extract the most likely predictions from this output distribution, making the proposed approach suitable for real-time applications. Empirically we find that our model i) achieves performance on par with or better than the state-of-the-art, ii) captures uncertainties and correlations inherent in pose estimation and iii) can exploit additional information at test time, such as multi-view consistency or body shape priors. CondiMen spices up the modeling of ambiguity, using just the right in- gredients on hand.

Abstract: Human mesh recovery is a fundamental and challenging task in computer vision. Existing image-based methods suffer from depth ambiguity due to the absence of explicit 3D contextual information. Conversely, video-based methods leverage multi-view input and temporal consistency to improve stability but struggle with capturing fine-grained spatial details and have high computational costs. To effectively combine the spatial precision of image-based techniques with the temporal robustness of video-based approaches, we propose a temporal Transformer framework augmented with the state-of-the-art image-based reconstruction model, Virtual Markers. Specifically, we introduce a novel disentanglement module designed to explicitly separate Virtual Markers into distinct pose and shape representations. Leveraging short-term temporal context, the proposed module enhances the consistency of body shape and pose coherence across frames, ensuring both spatial accuracy and computational efficiency. Experimental results demonstrate that the proposed method significantly enhances the performance and interpretability of virtual markers. Our model achieves state-of-the-art results on two widely used benchmarking datasets, outperforming previous image-based approaches across different evaluation metrics.