Request for code to build the SKEL model

[Kelly510]

Hi! Thank you for your wonderful work. I am wondering if you have released the code for establishing the SKEL model, e.g. how you find the corresponding markers between SKEL and BSM with the help of the OSSO model. If haven't, could you please release it?

[MarilynKeller]

Hi, so we did release the code to establish the correspondance between SMPL markers and BSM: https://github.com/MarilynKeller/SMPL2AddBiomechanics. SKEL mesh has the same body mesh topology as SMPL, so the markers correspond to the same vertices in SMPL and SKEL.

The SKEL was learned from the BioAMASS dataset, that we released too (https://skel.is.tue.mpg.de/). By considering the orientation of the SMPL limbs and the BSM bones. That code we are not releasing but it's just loading BioAmass and learning a linear regressors.

[Kelly510]

Thanks for your information!
I have a further question about converting SMPL into SKEL. Here is an example in AMASS dataset (D7 - walk to bow). It is converted from the SMPL model using the code in align_to_SMPL_seq.py but the result is unrealistic. Do you know the reason that could possibly cause this result?

[MarilynKeller]

The pose looks weird indeed, is this an anatomically feasible SMPL pose? It would be usefull to see both SKEL and SMPL

If the SMPL body looks realistic, it might be a SKEL fitting issue, sometimes it does not converge to the proper solution. Are you using the last version of the code? I made the fit more robust a couple of months ago.

Note that you can export a video from aitviewer through the upper left menu which makes it easier to see the sequence.

[Kelly510]

amass_Male2MartialArtsStances_c3d_D7_walk_to_bow.mp4

Here is the video visualized by aitviewer. The code is the latest version.

[MarilynKeller]

I see, thanks for the video, indeed alignment issue:).
I'd suggest you try processing the sequences in two batches instead of one.

[Kelly510]

Thank you for your suggestion. May I ask why processing in two batches is better? Currently the code just fixes the batch size and the actual number of batches depends on how long the sequence is.

[MarilynKeller]

Set -w to the problematic frame index to watch the alignment of this frame.
Set -B to half or the third of your sequence frame number.

python examples/align_to_SMPL_seq.py yourseq.npz -D -w 0 -B 500

If this doesn't help, you can fine-tune the alignment weights:
https://github.com/MarilynKeller/SKEL/blob/master/skel/alignment/config.yaml

[MarilynKeller]

Thank you for your suggestion. May I ask why processing in two batches is better? Currently the code just fixes the batch size and the actual number of batches depends on how long the sequence is.

The batches following the first one are initialized with the pose of the last batch last frame, which can help.

This crazy pelvis issue is a recurent problem, I regularize it but could not understand the cause so far...

Instead of the batching, another thing you could try is commenting out this line:

SKEL/skel/alignment/aligner.py

Line 154 in 832f9a5

poses_skel[:, :3] = poses_smpl[:, :3] # Global orient are similar between SMPL and SKEL, so init with SMPL angles

(I'm just thinking this could be causing the issue but can not check myself right now.)

Or you can also try to change it to:
poses_skel[:, 2] = poses_smpl[:, 2] #(only use the z axis of SMPL to initialize the pelvis orientation)

[Kelly510]

It seems that the original pose parameter is used during the fitting process? Typically the pose should be converted to continuous representation of 6 or 9 elements. I couldn't find the place of this transformation.