Difference between revisions of "User Guide for Multiple Depth Sensors Configuration"
m |
|||
Line 69: | Line 69: | ||
In practice it may not always be achievable due to available space size and configuration, mount positions, USB and/or power cord lengths, etc. Thus, sensor positions may slightly differ from ideal ones. Please look at the example from real mocap session. | In practice it may not always be achievable due to available space size and configuration, mount positions, USB and/or power cord lengths, etc. Thus, sensor positions may slightly differ from ideal ones. Please look at the example from real mocap session. | ||
+ | <center> | ||
+ | {| | ||
+ | |[[Image:Triple-depth-sensor-scene-config.png|thumb|Real-life Scene Configuration Expample|500px]] | ||
+ | |} | ||
+ | </center> | ||
=== Recording Calibration Video === | === Recording Calibration Video === |
Revision as of 09:01, 2 November 2014
Contents
System Requirements
iPi Recorder
- Computer (desktop or laptop):
- CPU: x86 compatible (Intel Pentium 4 or higher, AMD Athlon or higher, 2GHz), dual- or quad- core is preferable
- Operating system: Windows 10 / 8.1 / 8 / 7 (32-bit or 64-bit)
- USB: at least two USB 2.0 or USB 3.0 controllers
- For more info see USB controllers
- Important! Kinect 2 for Windows, Kinect for Xbox One requires USB 3.0 controller. Kinect SDK 2.0 supports only single sensor on one PC. Libfreenect2 supports multiple sensors on one PC.
- ExpressCard or eSATA slot (for laptops)
- Optional, but highly recommended. It allows to install external USB controller in case of compatibility issues between cameras and built-in USB controllers, or if all USB ports are in fact connected to a single USB controller
- Storage system: HDD or SSD or RAID with write speed:
- For Kinect 2 for Windows / Kinect for Xbox One sensors:
- 2 sensors — not less than 66,0 MByte/sec
- 3 sensors — not less than 99,0 MByte/sec
- For Microsoft Kinect sensors:
- 2 sensors — not less than 55,0 MByte/sec
- 3 sensors — not less than 82,5 MByte/sec
- For Kinect 2 for Windows / Kinect for Xbox One sensors:

- 2 or 3 depth sensors: Microsoft Kinect 2 for Windows / Microsoft Kinect for Xbox One, or Microsoft Kinect
- To choose see Depth Sensors Comparison

- Optional: active USB 2.0 extension cables
- Optional: Kinect for Xbox One floor mount
- Optional: Kinect Adjustable Tripod Stands
- Minimum required space: 2.5 m by 2.5 m (8 by 8 feet)
iPi Mocap Studio
- Computer (desktop or laptop):
- CPU: x86/x64 compatible (Intel Pentium 4 or higher, AMD Athlon or higher), dual- or quad- core is preferable.
- Operating system: Windows 10 / 8.1 / 8 / 7 (32-bit or 64-bit).
- Video card: DirectX 11 capable gaming-class graphics card.
- For more info see Cameras and Accessories#Video_Card.
- You can use GPU-z and CPU-z to detect exact version and features of your video card.

{{Note|Before you start working with multiple depth sensors, it is highly recommended to get appropriate results with single depth sensor configuration: User Guide for Single Depth Sensor Configuration.
Software Installation
iPi Recorder

Download and run the setup package of the latest version of iPi Recorder. You will be presented with the following dialog.

- Select needed components
- Read and accept the license agreement by checking appropriate checkbox
- Press the Install button to begin installation


- You can plug only one depth sensor to one USB controller. Single USB controller bandwidth is not enough to record from 2 sensors.
- You can plug not more than 2 Sony PS Eye cameras to one USB controller, otherwise you will not be able to capture at 60 fps with 640 x 480 resolution.
- For more info see USB controllers.
Upon installation is complete, iPi Recorder will launch automatically. Continue with user's guide to get a knowledge of using the software.
Components
If some of the components is already installed, it has no checkbox and is marked with ALREADY INSTALLED label. You should not install all optional components in advance, without necessity. All of them can be installed separately at later time. Components descriptions below contain corresponding download links.
- Microsoft .NET Framework 4.5.1 - Client. This is required component and cannot be unchecked.
This is basic infrastructure for running .NET programs. iPi Recorder is a .NET program.- Web installer: http://www.microsoft.com/en-us/download/details.aspx?id=40773
- Standalone installer: http://www.microsoft.com/en-us/download/details.aspx?id=40779
- Playstation3 Eye Webcam :: WinUSB Drivers Registration. Check if you plan to work with Sony PS Eye cameras.
Device drivers for PS Eye camera.
- (Windows 8, 8.1, 10) Microsoft Kinect 2:: MS Kinect SDK 2.0. Check if you plan to work with Kinect 2 for Windows or Kinect for Xbox One depth sensors, but do not plan to connect multiple Kinects to a single PC.
Device drivers and software libraries for Microsoft Kinect 2. Requires 64-bit Windows 8+ and USB 3.0.
- (Windows 7, 8, 8.1, 10) Microsoft Kinect :: MS Kinect SDK 1.8. Check if you plan to work with Microsoft Kinect depth sensors.
Device drivers and software libraries for Microsoft Kinect. Requires Windows 7 and later.
- iPi Recorder 4.x.x.x. This is required component and cannot be unchecked.
iPi Recorder itself.
iPi Mocap Studio
Download and run the latest setup package of iPi Mocap Studio. You will be presented with the following dialog:

All components are required for installation.

Other components are included with iPi Mocap Studio setup.

- Press the Install button to begin installation.
- You will be prompted to read and accept the license agreement(s) by checking corresponding checkbox.
- Press the Install button to begin installation.
- Upon installation is complete, you will be prompted to launch iPi Mocap Studio.
- As soon as iPi Mocap Studio launches, you will be prompted to enter your license key or start 30-days free trial period.
For more info about license protection see Licensing Policy. - Ensure that your graphics hardware is set to maximum performance with iPi Mocap Studio.
Recording Video from Multiple Depth Sensors
Environment
Minimum space requirements for depth sensor configurations depend on sensor model and number of sensors:
Configuration | Azure Kinect (WFOV mode) | Azure Kinect (NFOV mode) Kinect 2 (Kinect for Xbox One) |
1st gen. Depth Sensors Orbbec Astra (Pro) |
---|---|---|---|
Single Depth Sensor |
7 by 5 feet = 2 by 1.5 meters |
8 by 5 feet = 2.5 by 1.5 meters |
10 by 10 feet = 3 by 3 meters |
2 Depth Sensors (90-degrees config.) |
7 by 7 feet = 2 by 2 meters |
8 by 8 feet = 2.5 by 2.5 meters |
10 by 10 feet = 3 by 3 meters |
2 Depth Sensors (180-degrees config.) |
12 by 7 feet = 3.5 by 2 meters |
16 by 8 feet = 5 by 2.5 meters |
20 by 10 feet = 6 by 3 meters |
3+ Depth Sensors |
12 by 12 feet = 3.5 by 3.5 meters |
16 by 16 feet = 5 by 5 meters |
20 by 20 feet = 6 by 6 meters |
Maximum capture area is about 7 by 7 feet ( = 2 by 2 meters), for all sensor models and configurations.
The pictures below will help you to understand possible capture area and required space. Dimensions are different for Azure Kinect, Kinect 2 (Kinect for Xbox One) and 1st gen. depth sensors due to different field of view.
Azure Kinect in NFOV mode and Kinect 2 for Windows (Kinect for Xbox One)
Kinect 2 (Kinect for Xbox One) sensors and Azure Kinect sensors in NFOV mode (narrow-view mode) have very close view angles (less than Azure Kinect in WFOV mode, but greater than 1st gen. depth sensors and Orbbec Astra):
Azure Kinect in WFOV mode
Wide-view mode (WFOV mode) of Azure Kinect has extremely wide view angle:

First Generation Depth Sensor or Orbbec Astra (Pro)
If you're using outdated 1st generation depth sensor like Kinect v1 or Orbbec Astra (Pro) sensor which is almost the same as 1st gen. depth sensor:
Below in #Calibration section you will find information on 2 recommended sensors' mutual configuratioins.
Actor Clothing
Current version uses only depth information to track motions. So clothes requirements are:
- no restrictions on clothes colors (just avoid shiny fabrics)
- please use slim clothes to reduce noise in resulted animation
Recording Process
Please record a video using iPi Recorder application. It supports recording with Sony PS Eye cameras, depth sensors (Kinect) and DirectShow-compatible webcams (USB and FireWire).
iPi Recorder is a stand-alone application and does not require a powerful video card. You may choose to install it on a notebook PC for portability. Since it is free, you can install it on as many computers as you need.
Please run iPi Recorder and complete setup and background recording steps following the instructions: iPi Recorder Setup
Calibration
Calibration is a process of computing accurate camera positions and orientations from a video of user waving a small glowing object called marker (for color/color+depth cameras). This step is essential and required for multi-camera system setup.


Dual Depth Sensors Configurations
There are two possible arrangements of the two sensors:
- angle between sensors is between 60 and 90 degrees;
- angle between sensors in near to 180 degrees that means that sensors are placed opposite to each other.
First configuration
Second configuration
Triple Depth Sensors Configuration
Ideally, depth sensors should be placed at apexes of equilateral triangle with sides equal to about 6 meters.
In practice it may not always be achievable due to available space size and configuration, mount positions, USB and/or power cord lengths, etc. Thus, sensor positions may slightly differ from ideal ones. Please look at the example from real mocap session.
Recording Calibration Video
After you finished preparations and sensors setup you're ready to record the calibration video:
- Start recording by pressing the Start button.
- Go to the capture area with the flashlight.
- Move the board back and forth, slight tipping and banking, plus slight side-to-side rotation.
- Stop the recording by pressing the "Stop" button.
Please open the captured video in the iPi Studio and check the following:
- The floor (ground) is good visible on the depth maps from both sensors
- The calibration board is good visible on the depth maps most the time
- The calibration board is moving and has variety in distances to the sensors
- The amount of yellow points is relatively small
Processing Calibration Video
- Please make sure that the ground plane is determined correctly for both sensors after calibration video opening in the iPi Studio (to switch between sensors use buttons "Camera 1" and "Camera 2" in the top toolbar).
- After that it is convenient to turn off visualization of background points (View -> Hide Background).
- Set the beginning of Region of Interest (ROI) to the point where calibration plane is good visible in both sensors.
- Adjust the end of ROI:
- the length of ROI should be at leat 5 seconds (150 frames)
- ROI should contain good variety of horizontal view angles for the calibration plane
- Go to the Calibration tab and click "Calibrate Based on 3D Plane" button
- Please wait for the end of calibration process
- Evaluate the result:
- turn on "Show Depth From All Sensors" option in the "View" menu
- rotate the scene and check how 3D-points from different sensors fit with each other
- Save the result:
- go to the Scene tab
- click "Save scene..." button
- choose file name for your scene parameters
Recording Actor's Performance
After completing Setup and Background recording steps, press Record button to begin video recording. To stop recording, press Stop button.
Recommended Layout of an Action Video
- Enter the actor.
- Strike a T-pose.
- Action
T-Pose
As soon as recorder starts, go to the capture area and stand in a T-pose. After that you can act desired motions.

If you make several takes of one actor, we recommend to strike a T-pose in the beginning of each take.

Takes
Take is a concept originating from cinematography. In a nutshell, take is a single continuous recorded performance.
Usually it is a good idea to record multiple takes of the same motion, because a lot of things can go wrong for purely artistic reasons.
Iterations
A common problem with motion capture is “clipping” in resulting 3D character animation. For example, arms entering the body of animated computer-generated character. Many CG characters have various items and attachments like a bullet-proof vest, a fantasy armor or a helmet. It can be easy for an actor to forget about the shape of the CG model.
For this reason, you may need to schedule more than one motion capture session for the same motions. Recommended approach is:
- Record the videos
- Process the videos in iPi Mocap Studio
- Import your target character into iPi Mocap Studio and review the resulting animation
- Give feedback to the actor
- Schedule another motion capture session if needed
Ian Chisholm's hints on motion capture
Ian Chisholm is a machinima director and actor and the creator of critically acclaimed Clear Skies machinima series. Below are some hints from his motion capture guide based on his experience with motion capture for Clear Skies III.
Three handy hints for acting out mocap:
- Don’t weave and bob around like you’re in a normal conversation – it looks terrible when finally onscreen. You need to be fairly (but not completely) static when acting.
- If you are recording several lines in one go, make sure you have lead in and lead out between each one, i.e. stand still! Otherwise, the motions blend into each other and it’s hard to pick a start and end point for each take.
- Stand a bit like a gorilla – have your arms out from your sides:
Well, obviously not quite that much. But anyway, if you don’t, you’ll find the arms clip slightly into the models and they look daft.
If you have a lot of capture to do, you need to strike a balance between short and long recordings. Aim for 30 seconds to 2 minutes. Too long is a pain to work on later due to the fiddlyness of setting up takes, and too short means you are forever setting up T-poses.
Takes
Because motion capture is not a perfect art, and neither is acting, it’s best to perform multiple takes. I found that three was the best amount for most motion capture. Take less if it’s a basic move, take more if it’s complex and needs to be more accurate. It will make life easier for you in the processing stage if you signal the break between takes – I did this by reaching out one arm and holding up fingers to show which take it was.
Naming conventions
As it’s the same actor looking exactly the same each and every time, and there is no sound, and the capture is in lowres 320*200, you really need to name the files very clearly so that you later know which act, scene, character, and line(s) the capture is for.
My naming convention was based on act, scene, character, page number of the scene, line number, and take number. You end up with something unpleasant to read like A3S1_JR_P2_L41_t3 but it’s essential when you’ve got 1500 actions to record.
Processing Video from Two Depth Sensors
Processing of performance video is generally the same as for single Kinect solution: User Guide for Single Kinect Sensor Configuration#Processing Video from Depth Sensor.
The main difference is that after loading captured video into iPi Studio you should load your scene parameters:
- switch to the Scene tab
- click "Load scene..." button
- select file with scene parameters which has been saved during calibration process
That's all. After that you're ready for the processing:
- Position timeline slider to the frame where actor is in T-pose
- Adjust actor height using appropriate slider on tab “Actor”
- Select “Move” tool on toolbar.
- Move actor model to left or right to match roughly actor silhouette on video.
- etc.
Clean-up
Export and Motion Transfer
Animation export and motion transfer
Troubleshooting
Video Materials
- From Jimer Lins: Part 1 - Setting up your Kinects and Calibrating
- From Jimer Lins: Part 2 - Recording the Action
- From Jimer Lins: Part 3 - Processing the Recorded Action
- From Jimer Lins: see other parts