Shenmue Animation Debug Thread

OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
One small improvement: added some axis helpers to the viewport to make it easier to track forward, back, left, right, to act as a reference for working with the lookup between motn and mt5 bone numbers.

quality_of_life.JPG

And then we run into a problem with interpolation.

Screenshot_2020-11-07 Shenmoo.png

I started to fill in the table with linear interpolation to be able to fill in the table and not worry too much about the easing functions while we're working with debugging. The problem is that I'm an idiot and the table is missing some values that should be there, so I need to figure out what's causing that inside my function.

But a little bit of a bigger problem is that not every value has a first frame and end frame value, which kind of leaves what goes in between there up to interpretation. I tried to look at a few other animations, only to find that my animation parsing function breaks on a lot of animation, specifically around block 4. So to be able to contrast with other animations, I should probably take some more animation samples, and trace through them. That should give some other animations to compare with to see how the data is laid out in the table.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
Added some more debug tools to the Shenpoo repository. At this point LemonHaze's plugin might make a lot of this obsolete, but hopefully tools like this make it easier to analyze and break down the animations.

Screenshot_2020-11-12 Shenpoo.png

For the viewer I added a small feature that let's you select which frame of an animation you want to look at. Right now all that gets moved is root bone position, base y position, and the right foot and left foot targets. We're going to need to implement the solver for the feet, hands and torso. The idea behind this approach is that it allows for fine tune control to be able to take a given frame, solve for all of the rotations in the bones. And then be able to play that back as a normal animation.

Screenshot_2020-11-12 Shenpoo(1).png

For the block 4 thing i think it's "fixed". And the reason ended up being that the uh, the animation format is kind of stupid. Block 2 gives the number of in-between keyframes. And then block 3 specifically defines the key frame numbers. And then block 4 provides a list of bit flags that define how to read the block 5 values for each one of the frames.

The weird part is that the way they do this you'd think every frame declared would have a value. For instance if you have an animation with 37 frames, then you have an implicit frame for frame 0 and 36. You then declare 3 in-between frames for something like 10, 20, 30. Which gives you [ 0, 10, 20, 30, 36 ]. Block 4 then defines bit flags to give a value or easing property for each one of these values. The problem is that it doesn't. Sometimes in block 4 there are no bits defined for a frame, which means no values. Which means the frame might as well not be defined at all.

I have no idea why that is, or what's going on. All I can say is that the values that are exported to the table look correct. And that the script I wrote to isolate different parts of the animation in the game works based on this interpretation. So right now all I can do is assume that I we have all of the data that we're going to get out of the motn file.

The concerning part is that it's not enough for an animation. For axis that don't have anything defined, I think the correct approach is to use the default bone value, and just plug it in there, and it doesn't have an effect (highlighted in yellow). The part that I'm concerned about is the red parts. Where the game often doesn't define any start or end frames, but defined middle frames.

And the reason I think that is, is probably the game has cross-fading built into it's animation system. And what I mean by that is that the game's programmer / director probably wanted the movement to flow. So that one action leads to another. And probably specifically when using IK, you're not moving the character directly but a target. For somethings like the hands, if you have a target for the beginning and end frame on all of the animations, it would probably cause some jankiness.

I think what this means is the developers decides to take the option to have one animation transition into the next. So for the walking animation, the devs probably repeated the flipped version of the walk animation. Which might be where the missing values for the beginning and end are, if they calculate from the last keyframe of the previous animation, into the first keyframe of the next animation.

If this is really the case, it would imply a lot of suck on the aspect of making the tool. As I might need to provide a "previous animation", or "next animation" option to be able to calculate the missing frame values. What these means for the moment is that I will cheat and hard code in values leading into the next walk cycle to fill in each of the x,y,z values needed to calculate a position for a frame.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
I have two main concerns at the moment. The first is the spline/bezier interpolation. Not because it's important, but mostly because it's vexing. In terms of pragmatic application, linear interpolation is going to be fine, it's mostly the prevalence of the easing values in the animation. This combined with the fact that I'm stupid to interpret these values causes my brain to wonder back to this topic pretty often. Right now, it's not a high priority so I will continue to put it off.

The second issue is the confidence in the animation values themselves. Right now what we can say about the values being read from the animation is that they seem to match up with the animation format. Meaning that when we read through the values, we end up of at the end of each respective block, and things seem to "fit". If we have the wrong interpretation things would be way off. The some what concerning part about the animation values, is they don't seem to be as "full" as I would like. And it could be that's an intentional move by the developers for built-in blending.

What we can say about the values is that right now they definitely seem "good-enough". As they seem to fit in with the expected parameters of the model. Meaning the values seem to have the correct side of the body, seem to be about the right height, and generally seem to be realistic values for what you would expect. So right now I want to set these two issues aside and jump into interpretation and start replication the IK functions in a clean environment where we can control what's happening at each frame. That way we can also fine-tune and control the rotations for each frame.

Basically jumping ahead and doing this gives us some really easy debug tools for viewing the animations. And then we can use these to improve the two issues mentioned above, as it would give us something pretty easy to check against.

Screenshot_2020-11-14 Screenshot.png

As luck would have it there is an example program that is really close to what suites our needs. The live version is here: https://yosipy.github.io/Threejs_IK_Sample/ and the Github Repository is here: https://github.com/yosipy/Threejs_IK_Sample. Weirdly enough it's by a Japanese developer, so I don't know if they have some kind unhealthy relationship with IK or not, but that's kind of beside the point. This example program has targets that can be dragged and then are solved by a function for each frame rather than being baked into the animation function like with the MMD loader.

After playing around with the example, the movement seems to look pretty natural. So I'm going to see about trying to copy and paste incorporate as much of this example as possible into the Shenpoo repository to solve for the rotations at any given frame. And right now I'm also working on the opposite, to try and take the animation values from Shenpoo and play them on this rig directly to see how well they work.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
After a few quick tests I am not getting the same results between the sample rig and the Shenmue rig. In the case of the sample rig, the girl model bends her knee behind her for the first frame, which look correct. While the Shenmue rig seems to just have Ryo's leg straight and pulled back.

Right now there are two main differences between these two rigs. The first is scale. The original size of the sample rig is probably about the same size as the Shenmue rig, but then it's scaled up by 400. So probably adjusting the scale would be a good idea. And the skeleton helper looks nice, but if it's difficult to adjust the scale on the that, then it might be a good idea to take that away in favor of using a normal Skeleton Helper.

The other difference is the pre-rotation values on the Shenmue rig. What I think I might do on this is create a clone of the Ryo mesh and have two copies. One with the pre-rotation values as originally defined by the mt5 file, and then add the option to toggle visibility between them.

 
Last edited:
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
Last night I managed to apply these changes. I scaled down the sample rig to 1:1 from 400:1 to be able to compare apples to apples if needed. After doing that I checked the values for the solver once the scale was adjusted, and i still got the same different result. From this I decided to then go ahead and remove pre-rotation from the Ryo Mt5 rig. And that seems to have been the issue and I'm now getting reasonable results from the (CCD) IK solver.

ik_solver_fixed.JPG

I think what the issue is with pre-rotations, the IK solver has a hard time figuring out the min and max rotation limits for different joints. This is similar to the issue that I was running into for the MMD loader, in that IK worked with removing pre-rotations, but got really terrible results when trying to add in FK rotation.

In this case I think it's possible to actually take the opposite approach. With the MMD solver, that was kind of out of my hands in terms of creating data and passing it into the animation parser. In this case I'm still in the general context of the editor, which provides more flexibility for how to interpret and display the animation state.

And I think what I can do in this case is the opposite of the MMD loader. In that case I removed the model's pre-rotation and then attempted to solve FK rotations for the rig with the removed rotations. In this case I think I can make two models, one with pre-rotation and one without pre-rotation. I can use the non-rotation model to solve for IK and then get the rotation for the bones and apply that to the original rig. We'll see how that goes.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
shenoo_screenshot.JPG

I'm not sure if this went better than expected or not. What's going on here is that we have the Ryo model with removed pre-rotation on the bottom, and then the vanilla mt5 model above it with 50% opacity. What's going on here is that we're using the rig with the removed pre-rotation to solve for the IK position, and then we take that solved rotation and apply it to the original model.

The weird part about this is that the the solved IK rotation is bending the knee in the X-direction. How ever if we apply this rotation to the original model as-is, it actually causes the knee to rotation inwards as opposed to bending the knee back. In order to be able to replicate the position I actually had to apply the X-rotation from the solved model, to the Z-axis on the original model.

And this is the quirky part about working with pre-rotation, as I think there is effectively a 90 degree rotation somewhere in there which causes the orientation of the bone to get swapped. Which is probably why the MMD IK solver and this IK solver have problems solving for the original model, expecting the T-pose to have zero initial rotation.

In this case we at lease got a proof of concept, and we have confirmed that we can apply the solved rotations to the original model if needed. The main question I have right now is what the best approach to do that is. Right now I'm simply swapping the axis for the knee from a small amount of trial and error. But I think the correct approach is to multiply the solved angle by the rotation of either the parent bone, or all of the parent bones up to the root. We'll have to do a little more testing to see how that works out.

For now I'm going to jump ahead a little bit. I'm going to manually fill in some of the missing values that are not included in the current table and then see if we can record each on of the rotations at each from and then export the leg part of the animation as a normal FK animation. Once we have confirmed that mechanic works, we can go back and start fine tuning more of the details.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
Finally some good news. For a while I've been hypothesizing about the possibility of solving the IK for each frame and converting that into an FK animation. The problem is that talk is easy, action takes time. And with that I've managed to talk the walk animation, solve the rotation for the knees and thighs at each frame, and then export those to a file as a normal animation.


In doing this, I skipped over a lot of details to focus on the proof of concept and specifically getting these animations outside of the preview editor and into a file. In terms of things to touch up, we still have the hands, look at target, and the torso IK. We also have the spline interpolation, and the confidence in the animation values and the foot planting.

The main defining aspect about this approach to the animation is probably going to be the built-in blending. You can do blending with FK, but the approach is generally pretty different. Normally FK animations like to have a first and last frame, and then when switching between animations you cross-fade from one to the next. In terms of the IK values, a lot of the start and end values for the animation simply don't exist in favor or manually cross-fading from state to the next by taking advantage of the IK target system.

Right now I think I'll continue to test the walk animation by manually filling in any missing values to test that way. After that we can take another look at the spline interpolation, and confidence for the values.
 

LemonHaze

Server Admin
Administrator
Joined
Dec 25, 2018
Firstly, in my old GIF below, you can see that the hands definitely do have a full range of keyframes. The reason you're not seeing them is likely due to a parsing issue, there is a block of keyframe data which should be for the hands.

Shown below is one of those keyframe blocks for the hands, with another with a static position.

ryo_walk_lhand_rhand.gif


The main defining aspect about this approach to the animation is probably going to be the built-in blending.

This part I'm not so sure will represent a big problem. The blending that happens in-game is primarily for interpolating the changes of animation states, for example the animation/blending between a walk and a run and from slightly jogging to full speed running/sprinting. Some animations which are used in-game will not look the same, like the 'medium' jogging animation. In this case with the walk animation, it is a standalone animation. Meaning that the whole sequence described in the animation we're looking at now is the only thing that's being played out on the rig in-game. So no blending here.

Now that you've done the part where you've split the FK and IK rigs, it might be worth checking out those rotation keyframes and confirming you can replicate the result there.
 
OP
OP
Kion

Kion

DashGL
Joined
Oct 11, 2018
Location
dashgl.com
Making some notes about the issues that are on the table at the moment. This is a stub, so I'll come back and try to add more information and notes about each point.


1. Parsing Confidence [ ]


2. Spline Interpolation [ ]

3. Rotation Key Frames [ ]

4. Animation Blending [ ]

5. Foot Planting [ ]

6. Look At Target [ ]

7. Hands IK [ ]

8. Torso IK [ ]

9. IK Environment Collision [ ]
 
Last edited:
Top