question about cvting SFM(BFM without ear&neck) to FLAME

[lvZic]

I want to convert a SFM model to FLAME. I try to replace the value in BFM_to_FLAME_corr by BFM_front_idx (https://github.com/sicxu/Deep3DFaceRecon_pytorch/blob/master/BFM/BFM_front_idx.mat) which means the corresponding face vertexs between SFM and BFM. However, the resulting obj is weired and the modified BFM_to_FLAME_corr matrix has many zeors-rows, which is false in my opinion.
I wonder how the BFM_to_FLAME_corr can be obtained for cvting SFM model to FLAME ?

[TimoBolkart]

Hi, can you please specify what exactly you did? Just zero-ed out the matrix entries for the vertices outside the mask? What is the dimensionality of your adapted correspondence matrix? Can you also please show some of the results that you get with this?

One alternative is to output the complete BFM mesh with the BFM parameters that e.g., Deep3DFaceRecon provides (rather than just outputting the cropped mesh) and converting this complete mesh to FLAME.

[lvZic]

Hi, can you please specify what exactly you did? Just zero-ed out the matrix entries for the vertices outside the mask? What is the dimensionality of your adapted correspondence matrix? Can you also please show some of the results that you get with this?

One alternative is to output the complete BFM mesh with the BFM parameters that e.g., Deep3DFaceRecon provides (rather than just outputting the cropped mesh) and converting this complete mesh to FLAME.

What i want to do is creaing a SFM_to_FLAME_corr matrix , similar to BFM_to_FLAME_corr matrix. As I have a SFM model and want to cvt it to FLAME. So I just modifyed the corresponding face vertexs in BFM_to_FLAME_corr matrix between SFM and BFM from /BFM_front_idx.mat).
But the resulting SFM_to_FLAME_corrmatirx has many zeors-rows, which is weird in my opinion.

[malteprinzler]

I think I might have found a solution, maybe it is also useful for others:

You can use this script to generate a new BFM_to_FLAME_corr.npz file. For this to work make sure you copy over the files select_vertex_id.txt and BFM_front_face_buf.txt (taken from https://github.com/sicxu/Deep3DFaceRecon_pytorch)
BFM_front_face_buf.txt
select_vertex_id.txt

import numpy as np
from psbody.mesh import Mesh
import matplotlib.pyplot as plt
import scipy

cached_data = np.load('./data/BFM_to_FLAME_corr.npz', allow_pickle=True, encoding="latin1")
selected_idcs = np.loadtxt('./data/select_vertex_id.txt').astype(int)  #
selected_faces = np.loadtxt('./data/BFM_front_face_buf.txt').astype(int)
mesh_path = "PATH_TO_TEST_BFM_MESH_FOR_VALIDATION.obj"
out_cache_file = "./data/BFM_to_FLAME_corr_SFM.npz"

BFM2009_corr = cached_data['BFM2009_corr'].item()
mtx = BFM2009_corr["mtx"]
mtx = mtx[:, :mtx.shape[1] // 2]
mtx_selected = mtx[:, selected_idcs]

flame_mask = np.zeros(mtx.shape[0], dtype=bool)
flame_mask_ids = cached_data["FLAME_mask_ids"]
flame_mask[flame_mask_ids] = True
flame_selected_mask = ~(np.abs(1 - np.array(mtx_selected.sum(axis=1)).flatten()) > 0.01)
flame_mask = flame_mask & flame_selected_mask
flame_mask_ids = np.where(flame_mask)[0]

out_cached_data = dict(
    FLAME_mask_ids=flame_mask_ids,
    BFM2009_SFM_corr=dict(f_in=selected_faces, mtx=scipy.sparse.hstack((mtx_selected, mtx_selected*0)), f_out=BFM2009_corr["f_out"], use_normals=BFM2009_corr["use_normals"]),
    BFM2009_cropped_corr=cached_data["BFM2009_cropped_corr"],
    BFM2009_corr=cached_data["BFM2009_corr"],
    BFM2017_corr=cached_data["BFM2017_corr"],
)
np.savez(out_cache_file, **out_cached_data, allow_pickle=True)

# # verification
# mesh = Mesh(filename=mesh_path)
# flame_verts = mtx_selected.dot(mesh.v)
# filtered_verts = flame_verts[flame_mask_ids]
# plt.scatter(filtered_verts[:, 0], filtered_verts[:, 1])
# plt.show()

In the end, this should give you this file here:
https://drive.google.com/file/d/1YDI1mIXO2CW9OcOXNNtAxL2Z03By5UIm/view?usp=sharing

You can either run the script above yourself or just download the resulting file from the link. Just put it into the data folder. Now, all you have to do is some minor changes in mesh_convert.py:

    cached_data = np.load('./data/BFM_to_FLAME_corr_SFM.npz', allow_pickle=True, encoding="latin1")

    BFM2017_corr = cached_data['BFM2017_corr'].item()
    BFM2009_corr = cached_data['BFM2009_corr'].item()
    BFM2009_cropped_corr = cached_data['BFM2009_cropped_corr'].item()
    BFM2009_SFM_corr = cached_data['BFM2009_SFM_corr'].item()

    if (2*BFM_mesh.v.shape[0] == BFM2017_corr['mtx'].shape[1]) and (BFM_mesh.f.shape[0] == BFM2017_corr['f_in'].shape[0]):
        conv_mesh = convert_mesh(BFM_mesh, BFM2017_corr)
    elif (2*BFM_mesh.v.shape[0] == BFM2009_corr['mtx'].shape[1]) and (BFM_mesh.f.shape[0] == BFM2009_corr['f_in'].shape[0]):
        conv_mesh = convert_mesh(BFM_mesh, BFM2009_corr)
    elif (2*BFM_mesh.v.shape[0] == BFM2009_cropped_corr['mtx'].shape[1]) and (BFM_mesh.f.shape[0] == BFM2009_cropped_corr['f_in'].shape[0]):
        conv_mesh = convert_mesh(BFM_mesh, BFM2009_cropped_corr)
    elif (2*BFM_mesh.v.shape[0] == BFM2009_SFM_corr['mtx'].shape[1]) and (BFM_mesh.f.shape[0] == BFM2009_SFM_corr['f_in'].shape[0]):
        conv_mesh = convert_mesh(BFM_mesh, BFM2009_SFM_corr)
    else:
        print('Conversion failed - input mesh does not match any setup')
        return

and it should work.