AI-based Tracking

From iPi Docs
Jump to: navigation, search


Starting from version 4.6.0 the iPi Mocap Studio offers new features that are based on integration with 3rd party AI (artificial intelligence) libraries like NVIDIA Maxine AR SDK.

These features are marked as "AI-based" or simply "(AI)" on UI and aimed to help with auxiliary tasks like:

  • detect initial actor pose
  • automatically fix tracking errors
  • track feet, hands and head rotations

System Requirements

OSMS Windows 10 or 11 64-bit
RAM8GB or higher
GPUNVIDIA RTX 20/30/40 Series, or NVIDIA Quadro RTX or NVIDIA RTX professional GPUs
DRIVERNVIDIA Display Driver version 520.46 or higher

If your system is not compatible, you can hide all new UI elements that requires AI libraries via Hide all AI-based Features on UI checkbox in Backward Compatibility collapsible group box on Tracking tab:


3rd Party Libraries Installation

To be able to use AI-based features, you should download and install NVIDIA Maxine AR 0.8.2 runtime on your own. It can be done either by downloading appropriate version of AR 0.8.2 runtime from the following page [1], or via the following dialog automatically shown by iPi Mocap Studio when AI-based detectors are required but NVIDIA Maxine AR 0.8.2 runtime cannot be found in the system:

maxine-ar-0 8 2-required-dlg.png

Limitations of Current Version

Currently, AI-based features are only available for the following types of projects/videos:

  • Multi camera RGB video, single actor
  • Single or multi depth sensor video with color data, single actor

That is, if your project contains multiple actors or it is recorded with the aid of depth sensor(s) without color data (only depth maps, no RGB images) then you are unable to use new AI-based features. Support for multiple actors will be added in future versions.

How to Use

Detect Pose (AI)

This feature is available on two Tabs: Actor and Tracking.

actor-detect-pose-ai.png  tracking-detect-pose-ai.png

In both cases, AI pose detector is applied to all available RGB images in the current frame, after that detection results are combined from all cameras, post-processed and Inverse Kinematics (IK) is used to produce pose of 3D character. This functionality is very useful to set initial pose before tracking or to fix tracking errors.

Tip: You can visualize raw output of AI detector for each camera by selecting View > Raw AI Skeleton item in the main menu. Raw AI Skeleton is shown as green balls connected with yellow sticks. Switch between cameras to see results for each camera.

After that, you can apply Refit Pose in most cases to refine 3D character position based on iPi proprietary tracking algorithms. However, please remember that correct actor parameters must be specified before using of Refit Pose.

Detect Pose and Height (AI)

Detect Pose and Height (AI) is available on Actor tab as an item in drop-down menu under Auto Adjust Actor Pose button. This feature works similar to the previous one but in addition to pose detection, it estimates actor’s height. The accuracy of such detection depends on pose, cameras views, etc. In most cases, it is better to set height manually if you know it. However, you can give it a try and see if it works for you.


AI-based feet tracking

Now there are three options for feet tracking:

  1. Do not track. Feet are not tracked and have default position (zero rotation) in all frames. Corresponds to unchecked state of Enable foot tracking checkbox in previous versions of iPi Mocap Studio.
  2. Track. Proprietary iPi algorithms are used to track feet. Corresponds to checked state of Enable foot tracking checkbox in previous versions of iPi Mocap Studio.
  3. Track (AI-based). AI detectors are used to determine feet positions in each frame during tracking pass. First introduced in version 4.6.0.

In some cases, iPi proprietary algorithms work better for feet, but for other videos AI-based tracking can be really useful.

tracking-settings-feet 4 6 0.png  tracking-settings-feet 4 5 8.png
Tip: AI-based tracking is very noisy. For this reason, please do not forget to apply aggressive Jitter Removal for body parts tracked with the help of AI.

AI-based head tracking

Similar to feet tracking, head position can be tracked with the aid of iPi proprietary algorithms or based on AI detectors. Again, for some videos (especially depth ones) AI-based approach can be more accurate.

AI-based hands tracking

In addition to feet and head, AI-based tracking can be applied to hands. Currently, only rotations for hands is supported. Neither thumb nor fingers positions are tracked. Use key framing for fingers, as before. In addition, accuracy of hands rotations determined by AI-based algorithms depends on hands visibility in cameras.


Applying only AI-based tracking

When one chooses AI-based tracking for Head/Hands/Feet, it applies during standard Tracking Forward / Backward pass along with iPi incremental tracking for full body. But when tracking has already been performed and there is no need to re-track frames in ROI, one can apply only AI-based tracking for Head/Hands/Feet (depending on selected settings for them). To do it, click on down arrow in the right part of Track Forward/Backward buttons and choose AI-based Only item in drop-down menu. This could be helpful during experiments with different AI options because it allows applying changes fast.

Note: If AI-based tracking is not selected for any body part, or tracking is turned off for all individual body parts that requires AI-based tracking, then AI-based Only option is not available in the drop-down menu.

Automatic tracking errors fixing

AI-based pose detection can help fixing tracking errors:

  • Choose frame with erroneous pose after tracking
  • Click Detect Pose (AI) button to set pose roughly based on AI detectors
  • Then click Refit Pose button to refine detected pose

In most cases, it should fix pose in the selected frame. But what if there are many sequential frames with erroneous poses? First, it is worth to try re-tracking in different direction from the fixed pose. It could help. If not then new Auto Fix Pose (AI-based) tracking option should be applied.


This option applies Detect Pose (AI) followed by Refit Pose actions for each frame in Forward/Backward directions, starting from the current frame.

Note: This approach is applicable even when AI-based tracking is not selected for any body part.

Tracking with Auto-recover option

This feature is more intellectual implementation of #Automatic tracking errors fixing. It is regularly compares the result of standard tracking with tracking that uses AI-detected pose as initial guess. If AI-based result is better, it uses it in the current frame and retacks back until it can improve tracking result. So if tracking error occurs, it is automatically detected, and misdetected frames are fixed. In most cases, this provides better tracking result and allows to avoid propagation of tracking errors to the subsequent frames.

How to track with Auto-Recover option:

  • Just select Algorithm option
    • Fast with Auto-recover (AI-based)
    • or Standard with Auto-recover (AI-based)

Additional time is required to compare with AI-based result and re-tracking, but this may save a lot of time as you would not need manual error fixing.

Note: This approach is applicable even when AI-based tracking is not selected for any body part.