View source for AI-based Tracking

== Introduction ==

Starting from version '''4.6.0''' the [[iPi Mocap Studio]] offers new features that are based on integration with 3<sup>rd</sup> party AI (artificial intelligence) libraries like [https://github.com/NVIDIA/MAXINE-AR-SDK#readme NVIDIA Maxine AR SDK].

These features are marked as '''"AI-based"''' or simply '''"(AI)"''' on UI and aimed to help with auxiliary tasks like:
* detect initial actor pose
* automatically fix tracking errors
* track feet, hands and head rotations

== System Requirements ==

<table class="wikitable" style="font-size:smaller;min-width:50%">
<tr><th>OS</th><td>MS Windows 10 or 11 64-bit</td></tr>
<tr><th>RAM</th><td>8GB or higher</td></tr>
<tr><th>GPU</th><td>NVIDIA RTX 20/30/40 Series, or NVIDIA Quadro RTX or NVIDIA RTX professional GPUs</td></tr>
<tr><th>DRIVER</th><td>NVIDIA Display Driver version 520.46 or higher</td></tr>
</table>

''If your system is not compatible, you can hide all new UI elements that requires AI libraries via '''Hide all AI-based Features on UI''' checkbox in '''Backward Compatibility''' collapsible group box on '''Tracking''' tab:''
<center>[[File:hide-all-ai-based-features.png]]</center>

== 3rd Party Libraries Installation ==

To be able to use AI-based features, you should download and install [https://github.com/NVIDIA/MAXINE-AR-SDK#readme NVIDIA Maxine AR 0.8.2] runtime on your own. It can be done either by downloading appropriate version of AR 0.8.2 runtime from the following page [https://www.nvidia.com/broadcast-sdk-resources], or via the following dialog automatically shown by [[iPi Mocap Studio]] when AI-based detectors are required but [https://github.com/NVIDIA/MAXINE-AR-SDK#readme NVIDIA Maxine AR 0.8.2] runtime cannot be found in the system:
 
<center>[[File:maxine-ar-0 8 2-required-dlg.png]]</center>

== Limitations of Current Version ==

Currently, AI-based features are only available for the following types of projects/videos:
* Multi camera RGB video, single actor
* Single or multi depth sensor video with color data, single actor

''That is, if your project contains multiple actors or it is recorded with the aid of depth sensor(s) without color data (only depth maps, no RGB images) then you are unable to use new AI-based features. Support for multiple actors will be added in future versions.''

== How to Use ==

=== Detect Pose (AI) ===

This feature is available on two Tabs: '''Actor''' and '''Tracking'''.

<center><table border="0">
<tr><td>[[File:actor-detect-pose-ai.png]]</td><td>&nbsp;&nbsp;</td><td>[[File:tracking-detect-pose-ai.png]]</td></tr>
</table></center>
      
In both cases, AI pose detector is applied to all available RGB images in the current frame, after that detection results are combined from all cameras, post-processed and Inverse Kinematics (IK) is used to produce pose of 3D character. This functionality is very useful to set initial pose before tracking or to fix tracking errors.

{{Tip|You can visualize raw output of AI detector for each camera by selecting '''View > Raw AI Skeleton''' item in the main menu. ''Raw AI Skeleton'' is shown as green balls connected with yellow sticks. Switch between cameras to see results for each camera.}}

<center>[[File:view-raw-ai-skeleton.png]]</center>
 
After that, you can apply Refit Pose in most cases to refine 3D character position based on iPi proprietary tracking algorithms. However, please remember that correct actor parameters must be specified before using of Refit Pose.

=== Detect Pose and Height (AI) ===

'''Detect Pose and Height (AI)''' is available on '''Actor''' tab as an item in drop-down menu under '''Auto Adjust Actor Pose''' button. This feature works similar to the previous one but in addition to pose detection, it estimates actor’s height. The accuracy of such detection depends on pose, cameras views, etc. In most cases, it is better to set height manually if you know it. However, you can give it a  try and see if it works for you.

<center>[[File:actor-detect-pose-and-height-ai.png]]</center>

=== AI-based feet tracking ===

Now there are three options for feet tracking:
# '''Do not track'''. Feet are not tracked and have default position (zero rotation) in all frames. Corresponds to unchecked state of '''Enable foot tracking''' checkbox in previous versions of [[iPi Mocap Studio]].
# '''Track'''. Proprietary iPi algorithms are used to track feet. Corresponds to checked state of '''Enable foot tracking''' checkbox in previous versions of [[iPi Mocap Studio]].
# '''Track (AI-based)'''. AI detectors are used to determine feet positions in each frame during tracking pass. First introduced in version '''4.6.0.'''

''In some cases, iPi proprietary algorithms work better for feet, but for other videos AI-based tracking can be really useful.''

<center><table border="0">
<tr><td>[[File:tracking-settings-feet_4_6_0.png]]</td><td>&nbsp;&nbsp;</td><td>[[File:tracking-settings-feet_4_5_8.png]]</td></tr>
<tr><th>v.4.6.0</th><th></th><th>v.4.5.8</th></tr>
</table></center>

{{Tip|AI-based tracking is very noisy. For this reason, please do not forget to apply aggressive [[Automatic_Refinement_and_Filtering|Jitter Removal]] for body parts tracked with the help of AI.}}

=== AI-based head tracking ===

Similar to feet tracking, head position can be tracked with the aid of iPi proprietary algorithms or based on AI detectors. Again, for some videos (especially depth ones) AI-based approach can be more accurate.

=== AI-based hands tracking ===

In addition to feet and head, AI-based tracking can be applied to hands. Currently, only rotations for hands is supported. Neither thumb nor fingers positions are tracked. Use [[Hand_Keyframing|key framing]] for fingers, as before. In addition, accuracy of hands rotations determined by AI-based algorithms depends on hands visibility in cameras.

<center>[[File:tracking-settings-hands-ai.png]]</center>

=== Applying only AI-based tracking ===

When one chooses AI-based tracking for Head/Hands/Feet, it applies during standard Tracking Forward / Backward pass along with iPi incremental tracking for full body. But when tracking has already been performed and there is no need to re-track frames in ROI, one can apply only AI-based tracking for Head/Hands/Feet (depending on selected settings for them). To do it, click on down arrow in the right part of '''Track Forward/Backward''' buttons and choose '''AI-based Only''' item in drop-down menu. This could be helpful during experiments with different AI options because it allows applying changes fast.
 
<center>[[File:tracking-ai-based-only.png]]</center>

{{Note|If AI-based tracking is not selected for any body part, or tracking is turned off for all individual body parts that requires AI-based tracking, then '''AI-based Only''' option is not available in the drop-down menu.}}

=== Automatic tracking errors fixing ===

AI-based pose detection can help fixing tracking errors:
* Choose frame with erroneous pose after tracking
* Click '''Detect Pose (AI)''' button to set pose roughly based on AI detectors
* Then click '''Refit Pose''' button to refine detected pose

In most cases, it should fix pose in the selected frame. But what if there are many sequential frames with erroneous poses? First, it is worth to try re-tracking in different direction from the fixed pose. It could help. If not then new '''Auto Fix Pose (AI-based)''' tracking option should be applied.
 
<center>[[File:tracking-auto-fix-pose-ai.png]]</center>

This option applies '''Detect Pose (AI)''' followed by '''Refit Pose''' actions for each frame in Forward/Backward directions, starting from the current frame.

{{Note|This approach is applicable even when AI-based tracking is not selected for any body part.}}

=== Tracking with Auto-recover option ===

This feature is more intellectual implementation of [[#Automatic tracking errors fixing]].
It is regularly compares the result of standard tracking with tracking that uses AI-detected pose as initial guess. If AI-based result is better, it uses it in the current frame and retacks back until it can improve tracking result. So if tracking error occurs, it is automatically detected, and misdetected frames are fixed. In most cases, this provides better tracking result and allows to avoid propagation of tracking errors to the subsequent frames.
 
How to track with Auto-Recover option:
* Just select '''Algorithm''' option
** Fast with Auto-recover (AI-based)
** or Standard with Auto-recover (AI-based)

Additional time is required to compare with AI-based result and re-tracking, but this may save a lot of time as you would not need manual error fixing.
 
<center>[[File:tracking-ai-auto-recover-2.png]]</center>

{{Note|This approach is applicable even when AI-based tracking is not selected for any body part.}}