Refine
Year of publication
- 2025 (3)
Document Type
- Article (2)
- In Proceedings (1)
Language
- English (3)
Has Fulltext
- no (3)
Is part of the Bibliography
- no (3)
Institute
Reconstructing the surfaces of deformable objects from correspondences between a 3D template and a 2D image is well studied under Shape-from-Template (SfT) methods; however, existing approaches break down when topological changes accompany the deformation. We propose a principled extension of SfT that enables reconstruction in the presence of such changes. Our approach is initialized with a classical SfT solution and iteratively adapts the template by partitioning its spatial domain so as to minimize an energy functional that jointly encodes physical plausibility and reprojection consistency. We demonstrate that the method robustly captures a wide range of practically relevant topological events including tears and cuts on bounded 2D surfaces, thereby establishing the first general framework for topological-change-aware SfT. Experiments on both synthetic and real data confirm that our approach consistently outperforms baseline methods.
This article presents a new method for non-rigidly registering a 3D shape to 2D keypoints observed by a constellation of multiple cameras. Non-rigid registration of a 3D shape to observed 2D keypoints, i.e., Shape-from-Template (SfT), has been widely studied using single images, but SfT with information from multiple-cameras jointly opens new directions for extending the scope of known use-cases such as 3D shape registration in medical imaging and registration from hand-held cameras, to name a few. We represent such multi-camera setup with the generalised camera model; therefore any collection of perspective or orthographic cameras observing any deforming object can be registered. We propose multiple approaches for such SfT: the first approach where the corresponded keypoints lie on a direction vector from a known 3D point in space, the second approach where the corresponded keypoints lie on a direction vector from an unknown 3D point in space but with known orientation w.r.t some local reference frame, and a third approach where, apart from correspondences, the silhouette of the imaged object is also known. Together, these form the first set of solutions to the SfT problem with generalised cameras. The key idea behind SfT with generalised camera is the improved reconstruction accuracy from estimating deformed shape while utilising the additional information from the mutual constraints between multiple views of a deformed object. The correspondence-based approaches are solved with convex programming while the silhouette-based approach is an iterative refinement of the results from the convex solutions. We demonstrate the accuracy of our proposed methods on many synthetic and real data.
Camera pose is a very natural concept in 3D vision in the rigid setting. It is however much more difficult
to work with in deformable settings. Consequently, numerous deformable reconstruction methods simply ignore
camera pose. We analyse the concept of pose in deformable settings and prove that it is unconstrained with the
existing formulations, properly justifying the existing pose-less methods reconstructing structure only. We explain
this result intuitively by the impossibility to define an intrinsic coordinate frame to a general deforming object. The
proposed analysis uses the isometric deformation model and extends to the weaker models including conformality
and equiareality. We propose a novel prior to rescue camera pose estimation in deformable settings, which attributes
the deforming object’s dominant rigid-body motion to the camera. We show that adding this prior to any existing
formulation fully constrains camera pose and leads to elegant two-step solution methods, involving deformable
structure reconstruction using a base method in the first step, and absolute orientation or Procrustes analysis in
the second step. We derive the proposed approach for the template-based and template-less settings, respectively
implemented using Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM) as base methods,
and validate them experimentally, showing that the computed pose is qualitatively and quantitatively plausible.