Difference between revisions of "Motion Controllers"

From iPi Docs
Jump to: navigation, search
m
(http->httpS)
 
(49 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Note| This article is under construction and will be published in a few days. Meanwhile refer to v2 documentation:
 
* [http://wiki.ipisoft.com/Getting_started_with_motion_controllers User Guide]
 
* [http://wiki.ipisoft.com/Using_motion_controller_for_head_tracking Using for Head Tracking]
 
* [http://wiki.ipisoft.com/Using_Wii_Remote_with_Toshiba_Bluetooth_stack Wii Remote with Toshiba Drivers]
 
}}
 
 
 
 
iPi Motion Capture allows to use motion controllers in addition to cameras for improved tracking of smaller details.
 
iPi Motion Capture allows to use motion controllers in addition to cameras for improved tracking of smaller details.
 
* Wrist rotations can be tracked only with motion controllers due to low resolution of video.
 
* Wrist rotations can be tracked only with motion controllers due to low resolution of video.
Line 14: Line 7:
 
= Equipment =
 
= Equipment =
  
<table style="float:right"><tr>
+
[[iPi Recorder]] supports these models of motion controllers:
<td>[[Image:Psmove.jpg|thumb|PlayStation Move]]</td>
+
 
<td>[[Image:WiiRemote.jpg|thumb|Wii Remote with MotionPlus]]</td>
+
{| class="wikitable"
<td>[[Image:Wii_remote_plus.jpg|thumb|100px|Wii Remote Plus]]</td>
+
! Model !! Bluetooth !! Magnetometer !! Measurement Rate !! Battery Life / Charging !! Comments
 +
|-
 +
|
 +
[[Image:Psmove.jpg|border|right]]
 +
PlayStation Move (model CECH-ZCM1x, older)
 +
| Bluetooth 2.1 EDR
 +
| <center>+</center>
 +
| 120 Hz
 +
| 10 hours <br> Charged with mini-USB cable
 +
|
 +
* Barcode at the back label starts with lower digit like '''1''' or '''GS3'''
 +
* This is the only model with magnetometer (see note below table)
 +
* Lighting sphere can be used for scene calibration with a light marker
 +
|-
 +
|
 +
[[Image:Psmove.jpg|border|right]]
 +
PlayStation Move (model CECH-ZCM1x, newer)
 +
| Bluetooth 2.1 EDR
 +
| <center>&ndash;</center>
 +
| 120 Hz
 +
| 10 hours <br> Charged with mini-USB cable
 +
|
 +
* Barcode at the back label starts with higher digit like '''4'''
 +
* Lighting sphere can be used for scene calibration with a light marker
 +
|-
 +
|
 +
[[Image:ps-move-2.jpg|border|100px|right]]
 +
PlayStation Move (model CECH-ZCM2x)
 +
| ? <br> ''Tested with ASUS USB-BT400 adapter (Bluetooth 4.0)''
 +
| <center>&ndash;</center>
 +
| 1200 Hz <br> ''10 times faster than CECH-ZCM1x''
 +
| 10+ hours <br> ''No official number but battery is known to have greater capacity than CECH-ZCM1x'' <br> Charged with micro-USB cable
 +
|
 +
* Lighting sphere can be used for scene calibration with a light marker
 +
|-
 +
| [[Image:joy-con.jpg|120px|border|right]]
 +
Nintendo Joy-Con
 +
| Bluetooth 3.0
 +
| <center>&ndash;</center>
 +
| 200 Hz
 +
| 20 hours <br> Charged by attaching to Switch console or Joy-Con grip
 +
|
 +
* Most compact dimensions
 +
|-
 +
| [[Image:WiiRemote.jpg|border|right]]
 +
Wii Remote with Motion Plus
 +
| Bluetooth 2.1 EDR
 +
| <center>&ndash;</center>
 +
| 100 Hz
 +
| 20-30 hours <br>
 +
2x AA batteries
 +
|
 +
* Needs to be paired on each usage.
 +
* Batteries are consumed even when off. So take them out when device is not used for a long period.
 +
|-
 +
| [[Image:Wii_remote_plus.jpg|100px|border|right]]
 +
Wii Remote Plus (with Motion Plus integrated)
 +
| Bluetooth 2.1 EDR
 +
| <center>&ndash;</center>
 +
| 100 Hz
 +
| 20-30 hours <br>
 +
2x AA batteries
 +
|
 +
* Needs to be paired on each usage.
 +
* Batteries are consumed even when off. So take them out when device is not used for a long period.
 +
|}
 +
 
 +
Your PC should be equipped with a Bluetooth radio module to connect with motion controllers. If you do not have a built-in one, you'll need an USB Bluetooth adapter.
 +
 
 +
{{Important|Each motion controller requires specific version of Bluetooth protocol. Usually adapters supporting higher versions are also compatible with lower version devices. However, this is not always the case. For instance, many BT 3.0/4.0 adapters do not work with PS Move CECH-ZCM1x model which requires BT 2.1 EDR. So if you're buying an adapter specifically for use with motion controllers, it may be safer to get one supporting the exact version of Bluetooth required by a particular motion controller.}}
 +
 
 +
{{Tip|Prefer '''Class 1''' adapters as they have extended range of operation (up to 100m).}}
 +
 
 +
[[iPi Recorder]] does not limit the number of connected motion controllers. However, the hardware does. Most Bluetooth adapters can handle up to 4 PS Move controllers (sometimes more). iPi Software supports up to 3 controllers for each actor. Thus, additional Bluetooth adapter(s) may be required. Unfortunately, standard Windows Bluetooth drivers do not support multiple adapters. We used to recommend Motion in Joy drivers that allowed support for multiple adapters, but these drivers are now out of support and thus not available anymore. However, if you use [[Distributed Recording]], you can use multiple adapters (one adapter per computer).
 +
 
 +
{{Note|A magnetometer improves the accuracy of output orientation in the long run by preventing its drift around vertical axis. However, this comes with downsides. First, magnetometer introduces sometimes noticeable delay in the response of output orientation to actual motions. Second, magnetometer measurements depend on environment. Thus, for fast motions and areas with high elecromagnetic noise PS Move with magnetometer may not work well.}}
 +
 
 +
= Recording =
 +
 
 +
To get your motion controller in the list of devices, it must be connected to a PC via Bluetooth. But before it can connect to your Bluetooth adapter, the controller must be ''paired'' with it. For that, press the '''Add Device''' button in the toolbar, select device type you have, and then follow the instructions specific for this device.
 +
<center>[[File:recorder-add-controller.png|400px]]</center>
 +
 
 +
{{Tip|When pairing Joy-Con or newer PS Move model CECH-ZCM2x (with micro-USB connector) you'll see a notification from Windows about adding a new device.
 +
 
 +
[[File:PS_Move_-_Add_device_notification.png|400px|border|Notification on Windows 10]]
 +
 
 +
Click on it to bring up a system dialog and allow connection.
 +
 
 +
[[File:PS_Move_-_Pair_device_dialog.png|400px|border|Pair dialog on Windows 10]]}}
 +
 
 +
For PS Move and Joy-Con, pairing is one-time process unless you reconnect motion controller to another PC or gaming console. Each time you connect controller to another host, it should be paired. After pairing, the Bluetooth connection is established automatically each time you power on the motion controller. To power on PS Move, press the '''PS''' button, for Joy-Con &mdash; any button on a front side. To power off, hold down the '''PS''' button for a long time (about 10 seconds), for Joy-Con &mdash; just press '''SYNC''' button.
 +
 
 +
For Wii Remote, pairing should be done each time after Bluetooth connection is lost, powering off either the motion controller or the PC. To power off Wii Remote, hold down the '''Power''' button for a couple of seconds.
 +
 
 +
{{Note|The recent revisions of Wii Remote Plus controllers may not work properly using the standard Bluetooth drivers. For them, you can use [[Wii Remote with Toshiba Bluetooth stack|Toshiba Bluetooth stack]].}}
 +
 
 +
<center>[[File:recorder-psmove-in-device-list.png|400px]]</center>
 +
 
 +
After you have successfully paired the controllers with Bluetooth adapter, they should be visible in device list. Select them in addition to cameras and press the '''Record Video''' button. Motion controller is visualized as a rotating blue "brick" showing its orientation in space. Hover mouse on the controller's screen and you'll see some useful controls like battery indicator and view selection.
 +
<center>[[File:recorder-motion-controller-visualization.png|400px]]</center>
 +
 
 +
When controller is being used for the first time (or have not been used for a long period) a message is displayed over it saying it is not calibrated. ''Calibration'' of a motion controller (not confuse with calibration of cameras) is needed for more accurate measurements, so it is highly recommended to get through it every time you see this message. To run calibration process, just click on the message or press the '''CALIBRATE''' button in the on-screen controls.
 +
 
 +
After a motion controller is calibrated, you can use it in recording action videos.
 +
 
 +
{{Tip|You can use motion controller to start/stop recording (including background evaluation). Just hold PS Move's '''Start''' button, or JoyCon's '''Capture''' (left) or '''Home''' (right) button, or Wiimote's '''A''' button for a couple of seconds, until you feel a vibration.}}
 +
 
 +
{{Tip|Generally, there is no need to enable motion controllers when recording calibration videos, because their data will not be used in the calibration process. However, you can use PS Move's light sphere as a light marker for calibration.<br/>You can light it with different colors using on-screen controls for PS Move or by pressing corresponding buttons on PS Move itself. Buttons work even if PS Move is not selected for recording. White color usually works best for the calibration.<br/>It is not recommended to light PS Move when recording action videos as it can lead to more tracking errors.}}
 +
{| class="wikitable"
 +
|-
 +
! Buttons !! Color
 +
|-
 +
| Select + Circle || <div style="background:Red; color: White; text-align: center">Red</div>
 +
|-
 +
| Select + Square || <div style="background:Magenta; color: White; text-align: center">Magenta</div>
 +
|-
 +
| Select + Triangle || <div style="background:Green; color: White; text-align: center">Green</div>
 +
|-
 +
| Select + Cross || <div style="background:Blue; color: White; text-align: center">Blue</div>
 +
|-
 +
| Select + Move || <div style="background:White; color: Black; text-align: center">White</div>
 +
|-
 +
| Select + Start || <div style="background:Yellow; color: Black; text-align: center">Yellow</div>
 +
|-
 +
| Select + T(rigger) || <div style="background:Cyan; color: Black; text-align: center">Cyan</div>
 +
|-
 +
| Select + PS || Off
 +
|}
 +
 
 +
= Tips on action video =
 +
 
 +
<table border="0" style="float:right"><tr>
 +
<td>[[File:motion-controller-aiming.jpg|thumb|200px|Aiming at a camera]]</td>
 +
<td>[[File:motion-controller-standard-grip.jpg|thumb|200px|Standard grip]]</td>
 
</tr></table>
 
</tr></table>
  
[[iPi Recorder]] supports these models of motion controllers:
+
There is one main requirement:
* '''PlayStation Move'''
+
# At some moment during the record (for example, at the beginning), you should aim with (each) motion controller at one of the cameras. This is needed to correctly orient the motion controller within the scene coordinate space in [[iPi Mocap Studio]].
* '''Wii Remote''' with '''MotionPlus'''
+
 
* '''Wii Remote Plus''', which has MotionPlus integrated
+
Now, the tips:
 +
* For a more natural movement of hands you can attach motion controllers to outer side of hands with tape instead of holding.
 +
* The recommended location for a motion controller on the head is at the top, with controller's top (light sphere for PS Move) looking forward.
 +
* To aim at camera with on-head controller (which you can't see), move the head slowly within a reasonable range from assumed direction to a camera. Then you'll select a frame with the most appropriate direction in [[iPi Mocap Studio]].
 +
* Write down which motion controller corresponds to which body part, by their order in [[iPi Recorder]].
 +
* If you want to see a natural orientation of prop models like the sword or the pistol in [[iPi Mocap Studio]], hold the controller with its buttons facing the actor's face when an arm with a controller is stretched forward. We call it a "standard" grip (see picture).
 +
* You can use special accessories for motion controllers like [http://us.playstation.com/ps3/accessories/playstation-move-shooting-attachment-ps3.html shooting attachments]. In that case, the relative position of the controller to the prop is good known, so you just hold it as needed by the scenario (i.e. the '''requirement #2''' is not relevant for this case).
 +
* You can change the grip of the motion controller during the action. The [[iPi Mocap Studio]] allows you to process every frame range with the same grip independently.
 +
 
 +
= Tracking =
 +
 
 +
There is a tab in [[iPi Mocap Studio]] named '''Motion Controllers'''. It is visible only when open project contains data from one or more motion controllers. Numbering of controllers correspond to their order in [[iPi Recorder]] when a video was recorded.
 +
<center>[[File:motion-controllers-tab.png]]</center>
 +
 
 +
When a project with motion data is first created, you will see this data as one or more colored "bricks" floating somewhere under the actor's model. The color is used to differentiate motion controllers in the scene. You can change the color of each controller on the '''Motion Controllers''' tab.
 +
 
 +
== Aligning to the scene ==
 +
 
 +
To give more sense to this picture, for each controller you should specify a bone it is attached to. See the '''Attach to''' combobox. It contains '''LHand''', '''RHand''' and '''Head''' bones. When attached, the controller's "brick" is displayed near the corresponding bone of actor's model. For a nicer look, you have an option to replace the default brick with several prop models like sword and pistol.
 +
 
 +
<center>[[File:motion-controllers-attached.png]]</center>
 +
 
 +
To make some use of motion controller's data, it first needs to be matched with the scene coordinate space. Assuming that scene's ground plane is horizontal, controller's ''pitch'' and ''bank'' angles already correspond to the scene. However, ''heading'' angle of controller's orientation is measured from some "arbitrary" origin direction (in terms of the scene). For PlayStation Move it's always the North direction, for Wii Remote &mdash; a direction it was oriented to when recording started. To match coordinates, knowing the scene coordinates for some reference orientation of the controller is needed. That is where aiming at a camera helps.
 +
 
 +
{{Tip|When using a motion controller for the head, do not forget to disable head tracking when doing the main tracking process of the body. This will save you much time on processing.}}
 +
 
 +
So you've done with the body tracking, and have attached the controller to a corresponding bone. Now find some frame where the actor is aiming with this controller at some camera. Ensure the body position has been tracked in this frame. Select the target camera as the default, pressing corresponding toolbar button. Pressing the '''Match with Camera #N''' button does matching. After matching, the prop model for the controller should be oriented perfectly as it was intended in the video. Unless the actor did not pay attention to [[#Tips on action video|a proper grip]] of the controller, especially regarding it's bank rotation.
 +
 
 +
Also you can manually edit position and rotation of the controller with the '''Move''' tool. Activate the tool, click on the controller's model in the scene, and then move and rotate the model to correspond to a position and orientation of an actual controller in the video. This may be used instead of aiming and automatic matching with camera, but is usually less accurate.
 +
 
 +
{{Tip|If an actor did not aim controller at any camera specifically, look thoroughly through all the frames &mdash; it is very likely that at some point the actor aimed at some camera by accident, at least in horizontal direction (this is sufficient).}}
 +
 
 +
{{Tip|If you have attached a controller to the head far from the default "top" location (say, at head's side above the ear), then you may adjust its position '''before''' pressing '''Match with camera''' to improve matching accuracy.}}
 +
 
 +
{{Important|Alignment to the scene is required only once per project and motion controller. That is, you don't need to align again before each application of motion data. However, for better results it may be needed to re-align controller for different regions of a video because of the drifting of motion data with time.}}
 +
 
 +
== Applying motion data ==
 +
 
 +
And finally &mdash; applying the motion controller's data to the bone rotation. This makes sense only after the tracking of the body is complete. In case of several controllers, data from each of the controllers is processed separately in any convenient order.
  
Your PC should be equipped with a Bluetooth radio module to connect with motion controllers. If you do not have a built-in one, you'll need an USB Bluetooth adapter.  
+
First, determine your region of interest. Ensure that within this region of frames the actor maintains a fixed grip of the controller. Frames with different grip should be processed separately. Next, select some frame within the region where the orientation of actor's bone is well visible on video. Edit the orientation of model's bone with the '''Rotate''' tool to correspond to the video. When ready, press the '''Apply Motion Data''' button &mdash; and in all frames within ROI, the bone's orientation will be set automatically according to relative orientation of bone and controller in the current frame.
  
[[iPi Recorder]] does not limit the number of connected motion controllers. However, the hardware does. Usually, a single Bluetooth adapter can handle up to 4 PS Move controllers. This is sufficient for a single actor, but for multiple actors you may need more controllers. Thus, additional Bluetooth adapter(s) is required. Unfortunately, standard Windows Bluetooth drivers do not support multiple adapters. To use multiple Bluetooth adapters, it is necessary to [[PlayStation Move with Motion in Joy drivers|install Motion in Joy drivers]].
+
=== Recommended Workflow ===
  
For PS Move, an USB to mini-USB cable is also required to charge your motion controller and for pairing (see [[#Getting into Recorder|next section]]).
+
Basically, motion controllers' data should be applied after ''Tracking'' / ''Refine'' / ''Jitter Removal'' stages are performed. However, starting from ''Version 4.0.3'', motion controllers' data can also be used to improve ''Refine'' stage quality. If you apply motion controllers' data before ''Refine'' stage, hands and / or head orientations obtained from motion controllers will be used during ''Refine'' thus allowing to match model with video more accurately. The recommended workflow is thus the following:
 +
# Perform ''Tracking''.
 +
# Perform [[Manual Clean-up]] if necessary.
 +
# [[Motion_Controllers#Aligning_to_the_scene|Align controllers to the scene]] and [[Motion_Controllers#Applying_motion_data|Apply motion data]]
 +
# Perform ''Refine''
 +
# Perform ''Jitter Removal''
 +
# [[Motion_Controllers#Applying_motion_data|Apply motion data]]
  
{{Tip|Unless you already have Wii stuff, we strongly recommend to use PlayStation Move, because of its built-in magnetometer. In short, the presence of magnetometer gives a lot more stable orientation value. Without it, orientation tends to drift around vertical axis, and this leads to less accurate measurements.}}
+
=== Head Rotation ===
 +
By default on-head controller's rotation is applied to Head joint only. Select ''Distribute rotation between Head and Neck'' option to distribute rotation between Head and Neck joints. Head / Neck distribution setting slider will show. Bigger value means bigger part of rotation goes to Head.
 +
<center>[[File:head-motion-controller-head-and-neck-1.png|border]]</center>
  
{{Note|On '''Windows 8 and 8.1''', PS Move does not work properly using the standard Bluetooth drivers included with the OS. For these OS versions, it is recommended to [[PlayStation Move with Motion in Joy drivers|install Motion in Joy drivers]].}}
+
= Tutorials and demos =
  
{{Note|The recent revisions of Wii Remote Plus controllers may not work properly using standard Bluetooth drivers. For them, it is recommended to use [[Wii Remote with Toshiba Bluetooth stack|Toshiba Bluetooth stack]].}}
+
* A great sample of hand tracking from Seed Crystal Games studio:
 +
*:  https://www.youtube.com/watch?v=4mV7yITXyBU
 +
* PS3 Move Gloves for Hand by CManson:
 +
*: https://forum.ipisoft.com/viewtopic.php?f=3&t=6836
 +
* DIY solutions for attaching PS Move to a head:
 +
*: https://forum.ipisoft.com/viewtopic.php?f=3&t=7134
 +
* Video of pairing procedure for PS Move:
 +
*: http://vimeo.com/bassill/pairing-ps-move-in-recorder

Latest revision as of 03:51, 4 March 2021

iPi Motion Capture allows to use motion controllers in addition to cameras for improved tracking of smaller details.

  • Wrist rotations can be tracked only with motion controllers due to low resolution of video.
  • Though head can be tracked with camera, using a motion controller for that purpose greatly reduces the processing time and may provide better accuracy.

Essentially, a motion controller provides 3D orientation value which can be used to track rotations of a single bone it is attached to.

Equipment

iPi Recorder supports these models of motion controllers:

Model Bluetooth Magnetometer Measurement Rate Battery Life / Charging Comments
Psmove.jpg

PlayStation Move (model CECH-ZCM1x, older)

Bluetooth 2.1 EDR
+
120 Hz 10 hours
Charged with mini-USB cable
  • Barcode at the back label starts with lower digit like 1 or GS3
  • This is the only model with magnetometer (see note below table)
  • Lighting sphere can be used for scene calibration with a light marker
Psmove.jpg

PlayStation Move (model CECH-ZCM1x, newer)

Bluetooth 2.1 EDR
120 Hz 10 hours
Charged with mini-USB cable
  • Barcode at the back label starts with higher digit like 4
  • Lighting sphere can be used for scene calibration with a light marker
ps-move-2.jpg

PlayStation Move (model CECH-ZCM2x)

 ?
Tested with ASUS USB-BT400 adapter (Bluetooth 4.0)
1200 Hz
10 times faster than CECH-ZCM1x
10+ hours
No official number but battery is known to have greater capacity than CECH-ZCM1x
Charged with micro-USB cable
  • Lighting sphere can be used for scene calibration with a light marker
joy-con.jpg

Nintendo Joy-Con

Bluetooth 3.0
200 Hz 20 hours
Charged by attaching to Switch console or Joy-Con grip
  • Most compact dimensions
WiiRemote.jpg

Wii Remote with Motion Plus

Bluetooth 2.1 EDR
100 Hz 20-30 hours

2x AA batteries

  • Needs to be paired on each usage.
  • Batteries are consumed even when off. So take them out when device is not used for a long period.
Wii remote plus.jpg

Wii Remote Plus (with Motion Plus integrated)

Bluetooth 2.1 EDR
100 Hz 20-30 hours

2x AA batteries

  • Needs to be paired on each usage.
  • Batteries are consumed even when off. So take them out when device is not used for a long period.

Your PC should be equipped with a Bluetooth radio module to connect with motion controllers. If you do not have a built-in one, you'll need an USB Bluetooth adapter.

Important.png
Important! Each motion controller requires specific version of Bluetooth protocol. Usually adapters supporting higher versions are also compatible with lower version devices. However, this is not always the case. For instance, many BT 3.0/4.0 adapters do not work with PS Move CECH-ZCM1x model which requires BT 2.1 EDR. So if you're buying an adapter specifically for use with motion controllers, it may be safer to get one supporting the exact version of Bluetooth required by a particular motion controller.
Tip.png
Tip: Prefer Class 1 adapters as they have extended range of operation (up to 100m).

iPi Recorder does not limit the number of connected motion controllers. However, the hardware does. Most Bluetooth adapters can handle up to 4 PS Move controllers (sometimes more). iPi Software supports up to 3 controllers for each actor. Thus, additional Bluetooth adapter(s) may be required. Unfortunately, standard Windows Bluetooth drivers do not support multiple adapters. We used to recommend Motion in Joy drivers that allowed support for multiple adapters, but these drivers are now out of support and thus not available anymore. However, if you use Distributed Recording, you can use multiple adapters (one adapter per computer).

Note.png
Note: A magnetometer improves the accuracy of output orientation in the long run by preventing its drift around vertical axis. However, this comes with downsides. First, magnetometer introduces sometimes noticeable delay in the response of output orientation to actual motions. Second, magnetometer measurements depend on environment. Thus, for fast motions and areas with high elecromagnetic noise PS Move with magnetometer may not work well.

Recording

To get your motion controller in the list of devices, it must be connected to a PC via Bluetooth. But before it can connect to your Bluetooth adapter, the controller must be paired with it. For that, press the Add Device button in the toolbar, select device type you have, and then follow the instructions specific for this device.

recorder-add-controller.png
Tip.png
Tip: When pairing Joy-Con or newer PS Move model CECH-ZCM2x (with micro-USB connector) you'll see a notification from Windows about adding a new device.

Notification on Windows 10

Click on it to bring up a system dialog and allow connection.

Pair dialog on Windows 10

For PS Move and Joy-Con, pairing is one-time process unless you reconnect motion controller to another PC or gaming console. Each time you connect controller to another host, it should be paired. After pairing, the Bluetooth connection is established automatically each time you power on the motion controller. To power on PS Move, press the PS button, for Joy-Con — any button on a front side. To power off, hold down the PS button for a long time (about 10 seconds), for Joy-Con — just press SYNC button.

For Wii Remote, pairing should be done each time after Bluetooth connection is lost, powering off either the motion controller or the PC. To power off Wii Remote, hold down the Power button for a couple of seconds.

Note.png
Note: The recent revisions of Wii Remote Plus controllers may not work properly using the standard Bluetooth drivers. For them, you can use Toshiba Bluetooth stack.
recorder-psmove-in-device-list.png

After you have successfully paired the controllers with Bluetooth adapter, they should be visible in device list. Select them in addition to cameras and press the Record Video button. Motion controller is visualized as a rotating blue "brick" showing its orientation in space. Hover mouse on the controller's screen and you'll see some useful controls like battery indicator and view selection.

recorder-motion-controller-visualization.png

When controller is being used for the first time (or have not been used for a long period) a message is displayed over it saying it is not calibrated. Calibration of a motion controller (not confuse with calibration of cameras) is needed for more accurate measurements, so it is highly recommended to get through it every time you see this message. To run calibration process, just click on the message or press the CALIBRATE button in the on-screen controls.

After a motion controller is calibrated, you can use it in recording action videos.

Tip.png
Tip: You can use motion controller to start/stop recording (including background evaluation). Just hold PS Move's Start button, or JoyCon's Capture (left) or Home (right) button, or Wiimote's A button for a couple of seconds, until you feel a vibration.
Tip.png
Tip: Generally, there is no need to enable motion controllers when recording calibration videos, because their data will not be used in the calibration process. However, you can use PS Move's light sphere as a light marker for calibration.
You can light it with different colors using on-screen controls for PS Move or by pressing corresponding buttons on PS Move itself. Buttons work even if PS Move is not selected for recording. White color usually works best for the calibration.
It is not recommended to light PS Move when recording action videos as it can lead to more tracking errors.
Buttons Color
Select + Circle
Red
Select + Square
Magenta
Select + Triangle
Green
Select + Cross
Blue
Select + Move
White
Select + Start
Yellow
Select + T(rigger)
Cyan
Select + PS Off

Tips on action video

Aiming at a camera
Standard grip

There is one main requirement:

  1. At some moment during the record (for example, at the beginning), you should aim with (each) motion controller at one of the cameras. This is needed to correctly orient the motion controller within the scene coordinate space in iPi Mocap Studio.

Now, the tips:

  • For a more natural movement of hands you can attach motion controllers to outer side of hands with tape instead of holding.
  • The recommended location for a motion controller on the head is at the top, with controller's top (light sphere for PS Move) looking forward.
  • To aim at camera with on-head controller (which you can't see), move the head slowly within a reasonable range from assumed direction to a camera. Then you'll select a frame with the most appropriate direction in iPi Mocap Studio.
  • Write down which motion controller corresponds to which body part, by their order in iPi Recorder.
  • If you want to see a natural orientation of prop models like the sword or the pistol in iPi Mocap Studio, hold the controller with its buttons facing the actor's face when an arm with a controller is stretched forward. We call it a "standard" grip (see picture).
  • You can use special accessories for motion controllers like shooting attachments. In that case, the relative position of the controller to the prop is good known, so you just hold it as needed by the scenario (i.e. the requirement #2 is not relevant for this case).
  • You can change the grip of the motion controller during the action. The iPi Mocap Studio allows you to process every frame range with the same grip independently.

Tracking

There is a tab in iPi Mocap Studio named Motion Controllers. It is visible only when open project contains data from one or more motion controllers. Numbering of controllers correspond to their order in iPi Recorder when a video was recorded.

motion-controllers-tab.png

When a project with motion data is first created, you will see this data as one or more colored "bricks" floating somewhere under the actor's model. The color is used to differentiate motion controllers in the scene. You can change the color of each controller on the Motion Controllers tab.

Aligning to the scene

To give more sense to this picture, for each controller you should specify a bone it is attached to. See the Attach to combobox. It contains LHand, RHand and Head bones. When attached, the controller's "brick" is displayed near the corresponding bone of actor's model. For a nicer look, you have an option to replace the default brick with several prop models like sword and pistol.

motion-controllers-attached.png

To make some use of motion controller's data, it first needs to be matched with the scene coordinate space. Assuming that scene's ground plane is horizontal, controller's pitch and bank angles already correspond to the scene. However, heading angle of controller's orientation is measured from some "arbitrary" origin direction (in terms of the scene). For PlayStation Move it's always the North direction, for Wii Remote — a direction it was oriented to when recording started. To match coordinates, knowing the scene coordinates for some reference orientation of the controller is needed. That is where aiming at a camera helps.

Tip.png
Tip: When using a motion controller for the head, do not forget to disable head tracking when doing the main tracking process of the body. This will save you much time on processing.

So you've done with the body tracking, and have attached the controller to a corresponding bone. Now find some frame where the actor is aiming with this controller at some camera. Ensure the body position has been tracked in this frame. Select the target camera as the default, pressing corresponding toolbar button. Pressing the Match with Camera #N button does matching. After matching, the prop model for the controller should be oriented perfectly as it was intended in the video. Unless the actor did not pay attention to a proper grip of the controller, especially regarding it's bank rotation.

Also you can manually edit position and rotation of the controller with the Move tool. Activate the tool, click on the controller's model in the scene, and then move and rotate the model to correspond to a position and orientation of an actual controller in the video. This may be used instead of aiming and automatic matching with camera, but is usually less accurate.

Tip.png
Tip: If an actor did not aim controller at any camera specifically, look thoroughly through all the frames — it is very likely that at some point the actor aimed at some camera by accident, at least in horizontal direction (this is sufficient).
Tip.png
Tip: If you have attached a controller to the head far from the default "top" location (say, at head's side above the ear), then you may adjust its position before pressing Match with camera to improve matching accuracy.
Important.png
Important! Alignment to the scene is required only once per project and motion controller. That is, you don't need to align again before each application of motion data. However, for better results it may be needed to re-align controller for different regions of a video because of the drifting of motion data with time.

Applying motion data

And finally — applying the motion controller's data to the bone rotation. This makes sense only after the tracking of the body is complete. In case of several controllers, data from each of the controllers is processed separately in any convenient order.

First, determine your region of interest. Ensure that within this region of frames the actor maintains a fixed grip of the controller. Frames with different grip should be processed separately. Next, select some frame within the region where the orientation of actor's bone is well visible on video. Edit the orientation of model's bone with the Rotate tool to correspond to the video. When ready, press the Apply Motion Data button — and in all frames within ROI, the bone's orientation will be set automatically according to relative orientation of bone and controller in the current frame.

Recommended Workflow

Basically, motion controllers' data should be applied after Tracking / Refine / Jitter Removal stages are performed. However, starting from Version 4.0.3, motion controllers' data can also be used to improve Refine stage quality. If you apply motion controllers' data before Refine stage, hands and / or head orientations obtained from motion controllers will be used during Refine thus allowing to match model with video more accurately. The recommended workflow is thus the following:

  1. Perform Tracking.
  2. Perform Manual Clean-up if necessary.
  3. Align controllers to the scene and Apply motion data
  4. Perform Refine
  5. Perform Jitter Removal
  6. Apply motion data

Head Rotation

By default on-head controller's rotation is applied to Head joint only. Select Distribute rotation between Head and Neck option to distribute rotation between Head and Neck joints. Head / Neck distribution setting slider will show. Bigger value means bigger part of rotation goes to Head.

head-motion-controller-head-and-neck-1.png

Tutorials and demos