Difference between revisions of "User Guide for Multiple Depth Sensors Configuration"

Revision as of 09:01, 2 November 2014

System Requirements

iPi Recorder

• Computer (desktop or laptop):
  • CPU: x86 compatible (Intel Pentium 4 or higher, AMD Athlon or higher, 2GHz), dual- or quad- core is preferable
  • Operating system: Windows 10 / 8.1 / 8 / 7 (32-bit or 64-bit)
  • USB: at least two USB 2.0 or USB 3.0 controllers

    For more info see USB controllers

    

    Important! Kinect 2 for Windows, Kinect for Xbox One requires USB 3.0 controller. Kinect SDK 2.0 supports only single sensor on one PC. Libfreenect2 supports multiple sensors on one PC.

  • ExpressCard or eSATA slot (for laptops)

    Optional, but highly recommended. It allows to install external USB controller in case of compatibility issues between cameras and built-in USB controllers, or if all USB ports are in fact connected to a single USB controller

  • Storage system: HDD or SSD or RAID with write speed:
    • For Kinect 2 for Windows / Kinect for Xbox One sensors:
      • 2 sensors — not less than 66,0 MByte/sec
      • 3 sensors — not less than 99,0 MByte/sec
    • For Microsoft Kinect sensors:
      • 2 sensors — not less than 55,0 MByte/sec
      • 3 sensors — not less than 82,5 MByte/sec

Note: If your write speed is lower, you can use can use background subtraction mode. Alternatively, you can use compressed mode (that gives 3-5 times lower required write speed, but the CPU performance may become a bottleneck)

• 2 or 3 depth sensors: Microsoft Kinect 2 for Windows / Microsoft Kinect for Xbox One, or Microsoft Kinect
  • To choose see Depth Sensors Comparison

Note: For Microsoft Kinect for Xbox One you will need Microsoft Kinect Adapter cable

• Optional: active USB 2.0 extension cables
• Optional: Kinect for Xbox One floor mount
• Optional: Kinect Adjustable Tripod Stands
• Minimum required space: 2.5 m by 2.5 m (8 by 8 feet)

iPi Mocap Studio

• Computer (desktop or laptop):
  • CPU: x86/x64 compatible (Intel Pentium 4 or higher, AMD Athlon or higher), dual- or quad- core is preferable.
  • Operating system: Windows 10 / 8.1 / 8 / 7 (32-bit or 64-bit).
  • Video card: DirectX 11 capable gaming-class graphics card.
    • For more info see Cameras and Accessories#Video_Card.
    • You can use GPU-z and CPU-z to detect exact version and features of your video card.

{{Note|Before you start working with multiple depth sensors, it is highly recommended to get appropriate results with single depth sensor configuration: User Guide for Single Depth Sensor Configuration.

Software Installation

iPi Recorder

Important! Please unplug all cameras from computer before installation.

Download and run the setup package of the latest version of iPi Recorder. You will be presented with the following dialog.

1. Select needed components
2. Read and accept the license agreement by checking appropriate checkbox
3. Press the Install button to begin installation

Note: Most of the components require administrative privileges because they install device drivers or write to Program Files and other system folders. You will be presented with UAC prompts when appropriate during installation. If you plan to use iPi Recorder under user which has no administrative rights, you can pre-install other components separately using administrator's account.

Important!

1. You can plug only one depth sensor to one USB controller. Single USB controller bandwidth is not enough to record from 2 sensors.
2. You can plug not more than 2 Sony PS Eye cameras to one USB controller, otherwise you will not be able to capture at 60 fps with 640 x 480 resolution.

For more info see USB controllers.

Upon installation is complete, iPi Recorder will launch automatically. Continue with user's guide to get a knowledge of using the software.

Components

If some of the components is already installed, it has no checkbox and is marked with ALREADY INSTALLED label. You should not install all optional components in advance, without necessity. All of them can be installed separately at later time. Components descriptions below contain corresponding download links.

• Microsoft .NET Framework 4.5.1 - Client. This is required component and cannot be unchecked.
  This is basic infrastructure for running .NET programs. iPi Recorder is a .NET program.
  • Web installer: http://www.microsoft.com/en-us/download/details.aspx?id=40773
  • Standalone installer: http://www.microsoft.com/en-us/download/details.aspx?id=40779

• Playstation3 Eye Webcam :: WinUSB Drivers Registration. Check if you plan to work with Sony PS Eye cameras.
  Device drivers for PS Eye camera.
  • 64-bit OS: https://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x64.msi
  • 32-bit OS: https://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x86.msi

• (Windows 8, 8.1, 10) Microsoft Kinect 2:: MS Kinect SDK 2.0. Check if you plan to work with Kinect 2 for Windows or Kinect for Xbox One depth sensors, but do not plan to connect multiple Kinects to a single PC.
  Device drivers and software libraries for Microsoft Kinect 2. Requires 64-bit Windows 8+ and USB 3.0.
  • http://www.microsoft.com/en-us/download/details.aspx?id=44561

• (Windows 7, 8, 8.1, 10) Microsoft Kinect :: MS Kinect SDK 1.8. Check if you plan to work with Microsoft Kinect depth sensors.
  Device drivers and software libraries for Microsoft Kinect. Requires Windows 7 and later.
  • http://www.microsoft.com/en-us/download/details.aspx?id=40278

• iPi Recorder 4.x.x.x. This is required component and cannot be unchecked.
  iPi Recorder itself.

iPi Mocap Studio

Download and run the latest setup package of iPi Mocap Studio. You will be presented with the following dialog:

All components are required for installation.

Note: The installation of Microsoft .NET Framework 4.5.1 requires an Internet connection. If needed, you can download offline installer for Microsoft .NET separately, and run it before iPi Mocap Studio setup.

Other components are included with iPi Mocap Studio setup.

Note: Shell Extensions for Video and Project files are needed to show thumbnails and preview for iPi video and project files in Windows Explorer

1. Press the Install button to begin installation.
2. You will be prompted to read and accept the license agreement(s) by checking corresponding checkbox.

   

3. Press the Install button to begin installation.
4. Upon installation is complete, you will be prompted to launch iPi Mocap Studio.

   

5. As soon as iPi Mocap Studio launches, you will be prompted to enter your license key or start 30-days free trial period.
   For more info about license protection see Licensing Policy.

   

6. Ensure that your graphics hardware is set to maximum performance with iPi Mocap Studio.
   • Maximizing GPU performance in iPi Mocap Studio
   • Laptops with Switchable Graphics
   • Multiple GPUs

Recording Video from Multiple Depth Sensors

Environment

Minimum space requirements for depth sensor configurations depend on sensor model and number of sensors:

Maximum capture area is about 7 by 7 feet ( = 2 by 2 meters), for all sensor models and configurations.

The pictures below will help you to understand possible capture area and required space. Dimensions are different for Azure Kinect, Kinect 2 (Kinect for Xbox One) and 1st gen. depth sensors due to different field of view.

Azure Kinect in NFOV mode and Kinect 2 for Windows (Kinect for Xbox One)

Kinect 2 (Kinect for Xbox One) sensors and Azure Kinect sensors in NFOV mode (narrow-view mode) have very close view angles (less than Azure Kinect in WFOV mode, but greater than 1st gen. depth sensors and Orbbec Astra):

Side view		Top view

Azure Kinect in WFOV mode

Wide-view mode (WFOV mode) of Azure Kinect has extremely wide view angle:

Side view		Top view

Note: But please be aware that quality of depth map is better in narrow-view mode (NFOV mode). For this reason, if you have enough space, it is recommended to use NFOV mode for Azure Kinect sensors.

First Generation Depth Sensor or Orbbec Astra (Pro)

If you're using outdated 1st generation depth sensor like Kinect v1 or Orbbec Astra (Pro) sensor which is almost the same as 1st gen. depth sensor:

Side view		Top view

Below in #Calibration section you will find information on 2 recommended sensors' mutual configuratioins.

Actor Clothing

Current version uses only depth information to track motions. So clothes requirements are:

• no restrictions on clothes colors (just avoid shiny fabrics)
• please use slim clothes to reduce noise in resulted animation

Recording Process

Please record a video using iPi Recorder application. It supports recording with Sony PS Eye cameras, depth sensors (Kinect) and DirectShow-compatible webcams (USB and FireWire).

iPi Recorder is a stand-alone application and does not require a powerful video card. You may choose to install it on a notebook PC for portability. Since it is free, you can install it on as many computers as you need.

Please run iPi Recorder and complete setup and background recording steps following the instructions: iPi Recorder Setup

Calibration

Calibration is a process of computing accurate camera positions and orientations from a video of user waving a small glowing object called marker (for color/color+depth cameras). This step is essential and required for multi-camera system setup.

Important! Once you calibrated the camera system, you should not move your cameras for subsequent video shoots. If you move at least one camera, you need to perform calibration again.

Tip: We recommend to run calibration twice - before and after capture session. If any camera was moved during capture session, calibration made after the session can give you correct camera positions.

Dual Depth Sensors Configurations

There are two possible arrangements of the two sensors:

1. angle between sensors is between 60 and 90 degrees;
2. angle between sensors in near to 180 degrees that means that sensors are placed opposite to each other.

First configuration

Second configuration

Triple Depth Sensors Configuration

Ideally, depth sensors should be placed at apexes of equilateral triangle with sides equal to about 6 meters.

Ideal Configuration for Kinect 2

Ideal Configuration for Kinect

In practice it may not always be achievable due to available space size and configuration, mount positions, USB and/or power cord lengths, etc. Thus, sensor positions may slightly differ from ideal ones. Please look at the example from real mocap session.

Real-life Scene Configuration Expample

Recording Calibration Video

After you finished preparations and sensors setup you're ready to record the calibration video:

• Start recording by pressing the Start button.
• Go to the capture area with the flashlight.
• Move the board back and forth, slight tipping and banking, plus slight side-to-side rotation.
• Stop the recording by pressing the "Stop" button.

Please open the captured video in the iPi Studio and check the following:

• The floor (ground) is good visible on the depth maps from both sensors
• The calibration board is good visible on the depth maps most the time
• The calibration board is moving and has variety in distances to the sensors
• The amount of yellow points is relatively small

Processing Calibration Video

• Please make sure that the ground plane is determined correctly for both sensors after calibration video opening in the iPi Studio (to switch between sensors use buttons "Camera 1" and "Camera 2" in the top toolbar).

• After that it is convenient to turn off visualization of background points (View -> Hide Background).

• Set the beginning of Region of Interest (ROI) to the point where calibration plane is good visible in both sensors.

• Adjust the end of ROI:
  • the length of ROI should be at leat 5 seconds (150 frames)
  • ROI should contain good variety of horizontal view angles for the calibration plane

• Go to the Calibration tab and click "Calibrate Based on 3D Plane" button

• Please wait for the end of calibration process

• Evaluate the result:
  • turn on "Show Depth From All Sensors" option in the "View" menu
  • rotate the scene and check how 3D-points from different sensors fit with each other

• Save the result:
  • go to the Scene tab
  • click "Save scene..." button
  • choose file name for your scene parameters

Recording Actor's Performance

After completing Setup and Background recording steps, press Record button to begin video recording. To stop recording, press Stop button.

Recommended Layout of an Action Video

• Enter the actor.
• Strike a T-pose.
• Action

T-Pose

As soon as recorder starts, go to the capture area and stand in a T-pose. After that you can act desired motions.

Tip: Sometimes it is inconvenient or not possible to stand in T-Pose. You can do without it, but in this case initial alignment of actor model in iPi Mocap Studio will require more manual actions.

If you make several takes of one actor, we recommend to strike a T-pose in the beginning of each take.

Tip: We use tracking data from modern depth sensors for workflow improvements. You can get most of it if you follow recommendations described here.

Takes

Take is a concept originating from cinematography. In a nutshell, take is a single continuous recorded performance.

Usually it is a good idea to record multiple takes of the same motion, because a lot of things can go wrong for purely artistic reasons.

Iterations

A common problem with motion capture is “clipping” in resulting 3D character animation. For example, arms entering the body of animated computer-generated character. Many CG characters have various items and attachments like a bullet-proof vest, a fantasy armor or a helmet. It can be easy for an actor to forget about the shape of the CG model.

For this reason, you may need to schedule more than one motion capture session for the same motions. Recommended approach is:

• Record the videos
• Process the videos in iPi Mocap Studio
• Import your target character into iPi Mocap Studio and review the resulting animation
• Give feedback to the actor
• Schedule another motion capture session if needed

Ian Chisholm's hints on motion capture

Ian Chisholm is a machinima director and actor and the creator of critically acclaimed Clear Skies machinima series. Below are some hints from his motion capture guide based on his experience with motion capture for Clear Skies III.

Three handy hints for acting out mocap:

1. Don’t weave and bob around like you’re in a normal conversation – it looks terrible when finally onscreen. You need to be fairly (but not completely) static when acting.
2. If you are recording several lines in one go, make sure you have lead in and lead out between each one, i.e. stand still! Otherwise, the motions blend into each other and it’s hard to pick a start and end point for each take.
3. Stand a bit like a gorilla – have your arms out from your sides:
   
   
   
   Well, obviously not quite that much. But anyway, if you don’t, you’ll find the arms clip slightly into the models and they look daft.

If you have a lot of capture to do, you need to strike a balance between short and long recordings. Aim for 30 seconds to 2 minutes. Too long is a pain to work on later due to the fiddlyness of setting up takes, and too short means you are forever setting up T-poses.

Takes

Because motion capture is not a perfect art, and neither is acting, it’s best to perform multiple takes. I found that three was the best amount for most motion capture. Take less if it’s a basic move, take more if it’s complex and needs to be more accurate. It will make life easier for you in the processing stage if you signal the break between takes – I did this by reaching out one arm and holding up fingers to show which take it was.

Naming conventions

As it’s the same actor looking exactly the same each and every time, and there is no sound, and the capture is in lowres 320*200, you really need to name the files very clearly so that you later know which act, scene, character, and line(s) the capture is for.

My naming convention was based on act, scene, character, page number of the scene, line number, and take number. You end up with something unpleasant to read like A3S1_JR_P2_L41_t3 but it’s essential when you’ve got 1500 actions to record.

Processing Video from Two Depth Sensors

Processing of performance video is generally the same as for single Kinect solution: User Guide for Single Kinect Sensor Configuration#Processing Video from Depth Sensor.

The main difference is that after loading captured video into iPi Studio you should load your scene parameters:

• switch to the Scene tab
• click "Load scene..." button
• select file with scene parameters which has been saved during calibration process

That's all. After that you're ready for the processing:

• Position timeline slider to the frame where actor is in T-pose
• Adjust actor height using appropriate slider on tab “Actor”
• Select “Move” tool on toolbar.
• Move actor model to left or right to match roughly actor silhouette on video.
• etc.

Clean-up

Clean-up

Export and Motion Transfer

Animation export and motion transfer

Troubleshooting

Troubleshooting

Video Materials

• From Jimer Lins: Part 1 - Setting up your Kinects and Calibrating

• From Jimer Lins: Part 2 - Recording the Action

• From Jimer Lins: Part 3 - Processing the Recorded Action

• From Jimer Lins: see other parts