SCAPE: Shape Completion and Animation of People

TL;DR

• Pose Deformation
  • Input — mesh, align with template, different poses of a human
  • Params — pose deformation ( twist vector )
  • Output – mesh
• Body Deformation
  • Input — mesh, different individuals
  • Params — body deformation ( PCA )
  • Output – mesh

3. Acquiring and Processing Data Meshes

Range Scanning

Correspondence

Non-rigid Registration

Recovering the Articulated Skeleton

Data Format and Assumptions

4. Pose Deformation

4.1. Deformation Process

$$v_{k,j}^i=R_{\ell [k]}^i{Q}_k^i{\hat{v}}_{k,j}\text{qc}j=2,3$$

• ( input ) $Y^i$ — mesh in the data set containing different poses of a human
• ( input ) $p_k$ — triangle of the template
• ( input ) $x_{k,1},x_{k,2},x_{k,3}$ — points of triangle $p_k$
• ( input ) ${\hat{v}}_{k,j}^i=x_{k,j}-x_{k,1}\text{qc}j=2,3$ — triangle edges
• ( params ) $Q_k^i$ — $3\times 3$ linear transformation matrix
• ( input ) $R_{\ell }^i$ — rotation of the polygon’s rigid part in the articulated skeleton
• ( input ) $\ell [k]$ — body part associated with triangle $p_k$

However, the predictions for the edges in different triangles are rarely consistent. Thus, to construct a single coherent mesh, we solve for the location of the points $y_1,\dots ,y_M$ that minimize the overall least squares error:

$$\underset{y_1,\dots ,y_M}{\mathrm{argmin}}\sum _k\sum _{j=2,3}\text{norm}{R_{\ell [k]}^i{Q}_k^i{\hat{v}}_{j,k}-\text{pqty}{y}_{j,k}-y_{1,k}}^2$$

4.2. Learning the Pose Deformation Model

$$\begin{gathered} \begin{array}{rl}t& =\frac{\text{norm}\theta }{2\sin \text{norm}\theta }\text{bmqty}{m}_{32}-m_{23} \\ {m}_{13}-m_{31} \\ {m}_{21}-m_{12}\\ \text{qq}with\theta & ={\cos }^{-1}\text{pqty}\frac{\mathrm{tr}(M)-1}{2}\end{array} \end{gathered}$$

• ( analogous to $Q_k^i$ ) $M$ — any $3\times 3$ rotation matrix
• $m_{ij}$ — $M$’s $i$-th row and $j$-th column
• $t$ — twist for the joint angle, a 3D vector
  • direction of $t$ — axis of rotation
  • magnitude of $t$ — rotation amount

predicts the transformation matrices $Q_k^i$ as a function of the twists of its two nearest joints $\mathrm{\Delta }{r}_{\ell [k]}^i=\text{pqty}\mathrm{\Delta }{r}_{\ell [k],1}^i,\mathrm{\Delta }{r}_{\ell [k],2}^i$

$$q_{k,lm}^i=\text{vb}{a}_{k,lm}^T\text{vdot}\left[\begin{array}{c}\mathrm{\Delta }{r}_{\ell [k]}^i\\ 1\end{array}\right]l,m=1,2,3$$

• ( params ) $\text{vb}{a}_{k,lm}$ — $7\times 1$ regression vector

$$Q_k^i={\mathcal{Q}}_{\text{vb}{a}_k}\text{pqty}\mathrm{\Delta }{r}_{\ell [k]}^i$$$$\underset{\text{Bqty}{Q}_1^i,\dots ,Q_P^i}{\mathrm{argmin}}\sum _k\sum _{j=2,3}\text{norm}{R_k^i{Q}_k^i{\hat{v}}_{k,j}-v_{k,j}}^2+w_s\sum _{k_1,k_2\text{ adj}}I({\ell }_{k_1}={\ell }_{k_2})\text{vdot}\text{norm}{Q_{k_1}^i-Q_{k_2}^i}^2$$

• ( input ) $w_s=0.001\rho$
• ( input ) $\rho$ — resolution of the model mesh $X$
• $I(\cdot )$ — indicator function

5. Body-Shape Deformation

5.1. Deformation Process

$$v_{k,j}^i=R_{\ell [k]}^i{S}_k^i{Q}_k^i{\hat{v}}_{k,j}$$

• $S^i=\text{Bqty}{S}_k^i:k=1,\dots ,P$ — model the body deformation associated with subject $i$

5.2. Learning the Shape Deformation Model

PCA:

$$S^i={\mathcal{S}}_{U,\mu }\text{pqty}{\beta }^i=\overline{U{\beta }^i+\mu }$$

• ( params ) $S^i$ — $9\times N$

$$\underset{S^i}{\mathrm{argmin}}\sum _k\sum _{j=2,3}\text{norm}{R_k^i{S}_k^i{Q}_k^i{\hat{v}}_{k,j}^i-v_{k,j}^i}^2+w_s\sum _{k_1,k_2\text{ adj}}\text{norm}{S_{k_1}^i-S_{k_2}^i}^2$$

6. Shape Completion

7. Partial View Completion

8. Motion Capture Animation