Shenmue Animation Debug Thread

Switch said:
Good to hear it may be of use! Thanks for the transcription. The blurred kanji character looks to be 脇 waki which is the region of the armpit - perhaps something like "upper shoulder" might be appropriate.
Click to expand...

waki is the closest character that makes sense. The location of the label completely threw me off.

Also dropping this here to keep track, it's a debugger for Reicast/NullDC. But I don't know if it has a release or not.


Asked on Discord, this should be the repository/branch https://github.com/reicast/reicast-emulator/tree/div22/reios_std
 
Last edited:
Switch said:
Good to hear it may be of use! Thanks for the transcription. The blurred kanji character looks to be 脇 waki which is the region of the armpit - perhaps something like "upper shoulder" might be appropriate.
Click to expand...

1599059592990.png

Anything that looks like "kao" in your recently posted image?
 
Kion said:
This matches the number of bones and their names, which saves the time of having to track down each bone and figure out what it does.
Click to expand...

Incorrect. There are 25 bones listed in Switch's image.. I can see where you're coming from though, 0x25 = 37.. and 37 is the number of bones described in MOTN. :p

Also me and Phil have done the work of figuring out each bone for you! :D
 
LemonHaze said:
Also, any chance we could get the other page (the table of contents one)? It might have some nice clues in it! :D
Click to expand...

Here you go!

For example pages 6-19 shows entries for the movement of certain parts: neck & shoulder / elbows, arms, hands / fingers, waist, hip / thighs / knees, legs / pigtails, skirts etc.

D2MGvROU8AAj6Dd.jpg
 
I keep losing track of bones, names and the mapping for the virtual rig. So I might as well make a post to refer back to.

MT5 Bone IdMT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
0胸chest0
1頭headMOTN BONE Id
2MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
3MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
4MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
5MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
6右肩right_shoulder25
7右ひじright_elbow26
8右手right_hand30
9MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
10MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
11MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
12MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
13MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
14MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
15MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
16左肩left_shoulder31
17左ひじleft_elbow32
18左手left_hand36
19MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
20MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
21MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
22MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
23MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
24MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
25MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
26MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
27MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
28MT5 Bone Name (JP)MT5 Bone Name (EN)MOTN BONE Id
29腰hips1
30右ももright_thigh5
31右ひざright_knee-
32右足right_foot9
33右つえright_bridge-
34左ももleft_thigh12
35左ひざleft_knee-
36左足left_foot16
37左つえleft_bridge-
38右足IKright_foot_ik8
39左足IKleft_foot_ik15
40右手IKright_hand_ik29
41左手IKleft_hand_ik35

And then we have the IK chains. Which could also technically include the torso and the head look at target, but we can comeback to those. For the solver the easiest place to start is going to be the hands and feet. Using the MT5 bone id's, we have the following.

Code: 
this.iks.push({
    effector: 32, // right_foot
    iteration: 40,
    links: [{
        index: 31, // right_knee
        enabled: true,
        limitation: new THREE.Vector3(-1, 0, 0)
    }, {
        index: 30, // right_thigh
        enabled: true,
    }],
    maxAngle: Math.PI,
    target: 38 // right_foot_ik
});

this.iks.push({
    effector: 36, // left_foot
    iteration: 40,
    links: [{
            index: 35, // left_knee
            enabled: true,
            limitation: new THREE.Vector3(-1, 0, 0)
        },
        {
            index: 34, // left_thigh
            enabled: true,
        }
    ],
    maxAngle: Math.PI,
    target: 39 // left_foot_ik
});

this.iks.push({
    effector: 8, // right_hand
    iteration: 40,
    links: [{
            index: 7, // right_elbow
            enabled: true,
            limitation: new THREE.Vector3(-1, 0, 0)
        },
        {
            index: 6, // right_shoulder
            enabled: true,
        }
    ],
    maxAngle: Math.PI,
    target: 40 // right_hand_ik
});

this.iks.push({
    effector: 18, // left_hand
    iteration: 40,
    links: [{
            index: 17, // left_elbow
            enabled: true,
            limitation: new THREE.Vector3(-1, 0, 0)
        },
        {
            index: 16, // left_shoulder
            enabled: true,
        }
    ],
    maxAngle: Math.PI,
    target: 41 // left_hand_ik
});

For constrast we can also include the source for the CCD IK Solver: https://gitlab.com/dashgl/shenmoo/-/blob/master/public/mod/animation/CCDIKSolver.js. Specifically we the update function is from line 59 to 209, and then everything else in the file is the IK helper.

Right now I'm trying to think of what the best approach to reverse engineering the solving function would be. We need to trace through and find the specific function where the bone rotation is calculated for the IK target position. And then we also need a way to check which values result from that function to be able to check. And then from there we need to go our editor-clone and attempt to replicate the same values and functionality there.
 
Last edited:
I came across a bone ID table in one of the Dreamcast executables:

Code: 
UNID_MHARA          = 元 ‐ moto (base) 腹- hara (stomache)
UNID_C_MUNE         = breast
UNID_MUNE           = breast
UNID_L_MUNE         = breast
UNID_C_KAO          = face
UNID_KAO            = face
UNID_L_KAO          = face
UNID_WAKI_L         = clavicle L
UNID_C_KATA_L       = shoulder L
UNID_KATA_L         = shoulder L
UNID_HIJI_L         = elbow L
UNID_L_TE_L         = hand L / hand L effector maybe
UNID_TE_L           = hand L
UNID_WAKI_R         = clavicle R
UNID_C_KATA_R       = shoulder R
UNID_KATA_R         = shoulder R
UNID_HIJI_R         = elbow R
UNID_L_TE_R         = hand R / hand R effector maybe
UNID_TE_R           = hand R
UNID_C_KOSI         = waist
UNID_KOSI           = waist
UNID_L_KOSI         = waist L
UNID_HIP_L          = hip L
UNID_C_MOMO_L       = thigh L
UNID_MOMO_L         = thigh L
UNID_HIZA_L         = knee L
UNID_L_ASI_L        = leg L / foot L effector maybe
UNID_ASI_L          = leg L / foot L
UNID_TOE_L          = toe L
UNID_HIP_R          = hip R
UNID_C_MOMO_R       = thigh R
UNID_MOMO_R         = thigh R
UNID_HIZA_R         = knee R
UNID_L_ASI_R        = leg R / foot R effector maybe
UNID_ASI_R          = leg R / foot R
UNID_TOE_R          = toe R
UNID_C_KUBI         = neck
UNID_KUBI           = neck
UNID_L_KUBI         = neck
UNID_C_SIPPO        = tail
UNID_SIPPO_1        = tail
UNID_SIPPO_2        = tail
UNID_L_SIPPO        = tail
UNID_SIPPO_3_M      = tail
UNID_MIMI_L         = ear
UNID_MIMI_R         = ear
UNID_AGO            = chin
UNID_NO_USE         =
 
I kind of totally fell off the map in September, work got busy. I think I have a change to post and get back at doing something fun again. But one of the problems with being right at work, is that it generally means taking on more responsibility, so we'll see how this works out.

To recap, for Shenmue animations, we've kind of done the easy part which is reverse engineering the original file format to be able to read the content. What we've found with the original content is that it's basically in the exact form of a dope sheet. Specifically on the dope sheet we have a list of virtual bones divided into x, y, and z axis. And each axis can have rotation and/or position key frames assigned to it. The interpolation between key frames can either be linear or user a bezier curve to define a slope for easing.

If the game used only FK animations, then we would pretty much be done, as all we would need to is plug the key frame transformations into the parser and we would use something that closely resembles the animations in the game, if not an exact copy. Unfortunately for us, it looks like Yu Suzuki and the programmers of Shenmue were in try-hard mode and they used IK (specifically mixed FK-IK which is assumed for IK) probably for the purpose of making the combat actions look and feel more fluid.

That means that to be able to replicate the animations from the game, we need to replicate the IK solver from the game. Which honestly kind of sucks. While I haven't been active for the last month, I've been thinking about ways to approach this. One thing that I think would definitely help would be to make a dope sheet viewer for the data.


nMToqTo.png


It might seem like a waste of effort, but I haven't written anything dealing with bezier curves, so I'd like to make something that I can visual debug to be able to replication values that are being put into the solver for the in-between frames. I guess that alternatively we could generate a binary to remove all of the slope flags and values to create a version of the animation that uses linear interpolation in game. It would look robotic, but that would allow us to focus on the solver first, and then go back and add easing once we've confirmed how the solver works.
 
Last edited:
Okay, I guess we're going the route of replicating the IK-solver. So I went ahead and started to clean up my code to make changes easier. I was hoping to be able to use the CCD Solver provided with the the MMD loader, but I don't think the rotation format matches up. I tried to make a branch called quat to be able to solve for the rotation difference, but that didn't work out either.

The positive side of making our own solver means that I don't have to include the entirety of Threejs and all of it's modules. I simplified my code, and made a new repository in a format where it's easier to make more notes and debug tools for the animations.

Screenshot_2020-11-01 Shenmoo.png

Things that I'm working through right now:
- Trace through the mt5 model to make sure I label all of the bones
- Implement animation reader to be able to convert from binary to JSON to be able to display that
- Start trying to figure out the easing function to be able to create a table for every animation value at each frame

Repository: https://gitlab.com/dashgl/shenpoo
Live Version: https://dashgl.gitlab.io/shenpoo/
 
I'm still working with a somewhat constrained schedule. For now I'll keep working with baby steps in terms of implementing debug tools while working towards the wider objective of replicating the IK solver.

shenpoo.JPG

I went ahead and implemented something that I've been pretty lazy about in terms of filling in the names of the bones for the Ryo MT5 model. As I mostly focused on the bones that interact with motion and skipped over everything else. That everything else generally ended up being the fingers and jacket, but it's still nice to have a more complete image to work with.

In terms of approach, the MT5 model has a list of nodes with the position, rotation, scale, pointer to mesh, pointer to child and pointer to sibling. These nodes being pushed into a list give the bone number. And in the case of Shenmue the part of the mesh that is affected by the weight of the node is declared inside it. So what I did was pop one node at a time to see which part of the model disappeared and then worked my way backwards from the end of the model to the start of the model.

What I came across was when I popped node 02, the model dissappeared entirely. So I need to go back and look at the node numbers. I think I need to make an adjustment as to how I manage the nodes. I think Node 00 is the root node which is intended to be a point at (0,0,0) and then node 01 acts as the base of the body and is the parent node to the chest and the hips. I'll do a check to see if that's the case and then start on porting over the animation parser over to this repository.

Edit:

Small edit. I went ahead and checked to see if node 01 was the base node for the body and that seems to be the case. Originally I was setting the Y-position of the root node to 1.1 so that Ryo's feet would be even with the grid. Now I'm setting the Y-position of the body base node to 1.1, and with that we have a root node that's actually serving its purpose at (0,0,0).

Screenshot_2020-11-02 Shenmoo.png

Next step will be to port the animation parsing code (from binary to json object) over from the previous repository.
 
Last edited:
Kion said:
In terms of approach, the MT5 model has a list of nodes with the position, rotation, scale, pointer to mesh, pointer to child and pointer to sibling. These nodes being pushed into a list give the bone number. And in the case of Shenmue the part of the mesh that is affected by the weight of the node is declared inside it. So what I did was pop one node at a time to see which part of the model disappeared and then worked my way backwards from the end of the model to the start of the model.
Click to expand...
What are you doing this for? Myself and Phil have already done all of this for you, months and months ago now.. so I've made a pull request on the repo with all of the correct bone names and IDs.

gitlab.com

Fix bone reading code and use the latest MT5/MOTN bone IDs and names (!1) · Merge requests · DashGL / DC - Shenmue · GitLab

A threejs based tool for converting Shenmue animations to JSON
gitlab.com gitlab.com

1604322127561.png
 
Kion said:
I'm still working with a somewhat constrained schedule. For now I'll keep working with baby steps in terms of implementing debug tools while working towards the wider objective of replicating the IK solver.

View attachment 9541

I went ahead and implemented something that I've been pretty lazy about in terms of filling in the names of the bones for the Ryo MT5 model. As I mostly focused on the bones that interact with motion and skipped over everything else. That everything else generally ended up being the fingers and jacket, but it's still nice to have a more complete image to work with.

In terms of approach, the MT5 model has a list of nodes with the position, rotation, scale, pointer to mesh, pointer to child and pointer to sibling. These nodes being pushed into a list give the bone number. And in the case of Shenmue the part of the mesh that is affected by the weight of the node is declared inside it. So what I did was pop one node at a time to see which part of the model disappeared and then worked my way backwards from the end of the model to the start of the model.

What I came across was when I popped node 02, the model dissappeared entirely. So I need to go back and look at the node numbers. I think I need to make an adjustment as to how I manage the nodes. I think Node 00 is the root node which is intended to be a point at (0,0,0) and then node 01 acts as the base of the body and is the parent node to the chest and the hips. I'll do a check to see if that's the case and then start on porting over the animation parser over to this repository.

Edit:

Small edit. I went ahead and checked to see if node 01 was the base node for the body and that seems to be the case. Originally I was setting the Y-position of the root node to 1.1 so that Ryo's feet would be even with the grid. Now I'm setting the Y-position of the body base node to 1.1, and with that we have a root node that's actually serving its purpose at (0,0,0).

View attachment 9542

Next step will be to port the animation parsing code (from binary to json object) over from the previous repository.
Click to expand...
1604325352789.png

This is incorrect. The bone IDs are not identified like this and never have been. The actual bone ID itself comes from the first value within the node, after its been AND'ed with 255/0xFF. These are the correct bone IDs:

1604326399375.png

Please find answers to your headaches below:

gitlab.com

Fix bone reading code and use the latest MT5/MOTN bone IDs and names (!1) · Merge requests · DashGL / DC - Shenmue · GitLab

A threejs based tool for converting Shenmue animations to JSON
gitlab.com gitlab.com
 
IKBoneID.FootIKTarget_L @ 956898
Pos X:
Frame: 0 @956898
Val: 0xAE7F / -0.10150146484375 @[956902 / 0xE99E6]
----------------------------------------------------------

[1] - 0x149C5AE80+0xE99E6 // 0x7FF7FEB64866
read bp set here
1.png

[2] read with `setNewMOTNData()`
2.png

read bytes are written to 0xFEF41AF668
3.png

[3] value is now being expanded from f16 to f32 (`convertFourHalfFloatsToFloat`)
4.png
5.png

[4] value now has some time/interpolation related ops done on it and then written to a new space in memory (`readAndConvertPositionMOTNData`)
6.png
7.png

the new value (`0.10157251`) has now been used in the construction of a Vector3 (potentially Vector4), which is stored at 0x7FF7FE9AEE50

8.png
[5] the Vector3
9.png
is then translated
index.php

with the current transform matrix:
11.png
Vector3 still stored @ 0x7FF7FE9AEE50
12.png

kind of supplementary to the next point I make, here is the callee of translateWrapper:
13.png

I think this is simply creating a worldspace vector3 out of the previously created vector3.

[6]
the data around here, which I have made structures of (in which the one we're tracing is named tmpStr), represents the whole node structure and basically consists of offsets into the MOTN data itself, the expanded keyframe values and some other variables like node etc
14.png

[7] Setting a new breakpoint on the newly translated vec3 leads us to a known foot planting function.
This is pretty interesting as usually IK systems do foot planting after the usual animation system does its solving for the skeleton itself, for clarity, here is the foot planting function:
15.png

I've purposely clipped this rather short, as I've already done a lot of research on this function, including completely removing it and all animations still appear correctly, except Ryo no longer plants his feet on the ground:

I also tested this by removing the function from the executable before starting it and all of the results were the same, but for presentation purposes, I used Cheat Engine to make this change at runtime. It's important to note that from here on I simply make use of the disableFootPlanting offset shown in the screenshot to stop this function from reading any keyframe values.

So we'll skip this and move on.

[8] Now the game is reading the vector3 from this function:
16.png
Here, some pretty interesting stuff is happening.. the keyframe values keyframe->vec3 are now being used to create some rotations for the active node/bone.. this is 100% animation IK-related.

Stepping outside of this function, and applying some research, gives the following picture of everything:

17.png

This function calls MOTN::IKSolver 4 times and it seems to be arbitrarily transforming multiple nodes.
 
Last edited:
LemonHaze said:
What are you doing this for? Myself and Phil have already done all of this for you, months and months ago now.. so I've made a pull request on the repo with all of the correct bone names and IDs.
Click to expand...

With my testing before I was really focused on trying to find the most direct way possible to try and implement the animation. Which means I didn't pay attention to a lot of the details, and I'm making it a point to document the aspects I skipped over before. Thanks for the pull request, I went ahead and approved it.

LemonHaze said:
I win!

Click to expand...

Congratulations!

LemonHaze said:
This is incorrect. The bone IDs are not identified like this and never have been. The actual bone ID itself comes from the first value within the node, after its been AND'ed with 255/0xFF. These are the correct bone IDs:
Click to expand...

Again, thanks for the pull request. I think you have definitely mentioned this before. In the NinjaGD, the flags attribute in the node is used for flags, so something in my brain didn't quite click that this attribute had been repurposed.
 
One thing that I've been skipping over the is easing function. And for now I think it might be a good idea to be lazy and continue to skip over it. The debug tool I want to implement is a table that will take the values read from the motn file, solve for all of the in-between frames and then display those as a table. A linear interpolation should give us values that are close to what the game looks like, and for the moment they should be close enough while working on the animation implementation.

So the table is the next short-term objective to work towards. After that the focus will really be on how to solve the rotations for the joints in the IK target chains. And then from there figuring out how to implement those as an animation.
 
Screenshot_2020-11-05 Shenmoo.png

Still trying to work on steps that aren't too time consuming. Last night I went ahead and wrote a script that would write the animation values to a table to make it easier to debug. I think I have some small quality of life improvements to make on the style of the table.

- Add (short) bone names to the MOTN column.
- Add FK or IK to the type column
- Use dashed lines for x, y rows to make it easier to distinguish bones
- Use bold font for key frame values
- Use underline font to show when easing values are present
- Cross out rows that have no values
- Fill in the MT5 reference bone

After that I guess we should go ahead and fill in the table. For now it should be good enough to use linear interpolation. We can come back and use bezier easing if we figure out how to interpret that. Once we fill in the values, we'll then have to think of strategy for how to approach solving IK.
 
You must log in or register to reply here.
Back
Top