Problem Overview:

I am working on motion transfer from video to an FBX avatar (e.g., student.fbx) using React Three Fiber (R3F). I have completed pose estimation, joint correction, and mapping of necessary joints. Some end-effectors (e.g., hands, feet, head top) are missing, and I am keeping their parent-child bone relative rotations constant.

Input:

World-space joint positions (Spine as root)

Output:

Relative rotations for each bone in the skeletal hierarchy for accurate skeletal animation

Context:

The FBX avatar’s skeleton must be driven by these computed rotations, and the model is being rendered in R3F.

Some joints are missing, so their relative rotations are assumed constant.

Data Provided:

1. Joint Positions (world coordinates):

positions = {
    'Hips': np.array([0.00094648, -0.00167672, 0.00126527]),
    'Spine': np.array([-0.00342144, -0.23813844, 0.00518973]),
    'Chest': np.array([-0.00778935, -0.47460017, 0.00911419]),
    'Neck': np.array([-0.01215727, -0.71106189, 0.01303866]),
    'Head': np.array([-0.01652518, -0.94752361, 0.01696312]),
    'LeftShoulder': np.array([0.14024669, -0.45378777, -0.02814714]),
    'LeftArm': np.array([0.18454467, -0.29012263, -0.13864663]),
    'LeftForeArm': np.array([0.08727895, -0.38098565, -0.25202304]),
    'RightShoulder': np.array([-0.15582539, -0.49541256, 0.04637553]),
    'RightArm': np.array([-0.20083366, -0.19640198, 0.00503132]),
    'RightForeArm': np.array([-0.29409334, -0.01089536, -0.12074786]),
    'LeftHip': np.array([0.09170869, -0.00317237, 0.02767152]),
    'LeftUpLeg': np.array([0.07843398, 0.41216615, 0.01524313]),
    'LeftLeg': np.array([0.04706472, 0.63266933, 0.38847083]),
    'RightHip': np.array([-0.08981574, -0.00018107, -0.02514099]),
    'RightUpLeg': np.array([-0.0386166, 0.33015436, -0.01318303]),
    'RightLeg': np.array([-0.07297755, 0.70644695, 0.11082241])
}

2. Bone Hierarchy:

bone_hierarchy = {
    'Hips': 'Spine',
    'Chest': 'Spine',
    'Neck': 'Chest',
    'Head': 'Neck',
    'HeadTop': 'Head',
    'LeftShoulder': 'Chest',
    'LeftArm': 'LeftShoulder',
    'LeftForeArm': 'LeftArm',
    'LeftHand': 'LeftForeArm',
    'RightShoulder': 'Chest',
    'RightArm': 'RightShoulder',
    'RightForeArm': 'RightArm',
    'RightHand': 'RightForeArm',
    'LeftHip': 'Hips',
    'LeftUpLeg': 'LeftHip',
    'LeftLeg': 'LeftUpLeg',
    'LeftFoot': 'LeftLeg',
    'RightHip': 'Hips',
    'RightUpLeg': 'RightHip',
    'RightLeg': 'RightUpLeg',
    'RightFoot': 'RightLeg'
}

Key Challenges:

Relative Rotation Computation:

How do I compute the rotation of each bone relative to its parent (e.g., LeftArm relative to LeftShoulder) from world-space joint positions?

Should I compute direction vectors between parent and child and then derive quaternions (e.g., using Quaternion.fromUnitVectors())?

Coordinate System Alignment:

The joint positions are in world space, but the rotations need to be local to the parent’s coordinate frame.

What’s the best approach to correctly align these coordinate systems when computing rotations?

Handling Missing Joints:

Since some joints (e.g., LeftHand, RightFoot, HeadTop) are missing, is it acceptable to keep their relative rotations constant (e.g., inheriting the parent's rotation), or is there a better workaround?

What I Have Tried:

Loaded the FBX Model & Skeleton

Used FBXLoader to import the model.

Attached SkeletonHelper to visualize the bones.

Retrieved Joint Positions in World Space

Joint positions are stored as { jointName: [x, y, z] } in JSON.

Computed Relative Rotations

Used the parent and child joint positions to compute direction vectors.

Aligned bones using setFromUnitVectors(referenceUp, direction).

Applied rotations using slerp() for smooth blending.

Code Snippet (React Three Fiber - Bone Rotation Updater)

import React, { useEffect, useState, useRef } from 'react';
import { useFrame } from '@react-three/fiber';
import * as THREE from 'three';
import { FBXLoader } from 'three/examples/jsm/loaders/FBXLoader';

const BoneRotationUpdater = ({ modelPath, scale, position, rotation }) => {
    const [model, setModel] = useState(null);
    const meshRef = useRef();

    useEffect(() => {
        const loader = new FBXLoader();
        loader.load(modelPath, (loadedModel) => {
            loadedModel.scale.set(scale, scale, scale);
            loadedModel.position.set(...position);
            loadedModel.rotation.set(...rotation);
            setModel(loadedModel);
        }, undefined, console.error);
    }, [modelPath, scale, position, rotation]);

    useFrame(() => {
        if (!model) return;
        model.traverse((child) => {
            if (child.isBone) {
                // Compute direction vector from parent to child
                const parentPosition = new THREE.Vector3();
                const childPosition = new THREE.Vector3();
                child.parent?.getWorldPosition(parentPosition);
                child.getWorldPosition(childPosition);

                const direction = new THREE.Vector3().subVectors(childPosition, parentPosition).normalize();
                const referenceUp = new THREE.Vector3(0, 1, 0);
                const computedRotation = new THREE.Quaternion().setFromUnitVectors(referenceUp, direction);
                child.quaternion.slerp(computedRotation, 0.1);
            }
        });
    });

    return <group ref={meshRef} />;
};
export default BoneRotationUpdater;

Questions:

What’s the best method to convert world-space joint positions into relative bone rotations for an FBX avatar in React Three Fiber?

Are there any standard libraries, tutorials, or code examples that demonstrate this process, particularly with missing end-effectors?

How do I ensure proper coordinate system alignment when computing and applying these rotations?