How to Build Synced Multi-Camera Video

From iPi Docs
Jump to: navigation, search

Introduction

To process video recorded by action cameras (or other standalone camcorders) in iPi Mocap Studio you have to create multi-camera .iPiVideo file from separate MP4/MOV/AVI/MPEG video files recorded by these cameras:

build-combined-video-workflow-schema.png

Preparations

First of all, copy MP4/MOV/AVI/MPEG video files from each camera to computer.

There are two hints here:

  1. It's better to use microSD-card reader to copy files because it provides much higher speed of this operation in comparison with USB-connection.
  2. To distinguish files from different cameras it is convenient to create a sub-folder for each camera. And name of such sub-folder should contain label/nickname of corresponding camera.
    For example: if you have cameras with labels/names "cam1", "cam2", "cam3" and "cam4" then you can create the following structure of folders and files on your local disk:
YYYY-MM-DD-session-name
   cam1
      video files from "cam1" camera
   cam2
      video files from "cam2" camera
   cam3
      video files from "cam3" camera
   cam4
      video files from "cam4" camera
Alternatively you can rename video files right after copying (or even during copying) to local disk while you still remember which camera has produced those files. Such approach is used in the following sample: Sample Videos and Projects (four GoPro).

Overall Procedure

  • Run iPi Recorder and click Build Combined Video button in the toolbar at the home screen.
build-combined-video-tool-button.png
  • For each camera:
    • add video from this camera by clicking Add Video button in the toolbar — see mark (1) on screenshot below
    • choose this camera by clicking on the image from this camera — (2)
    • in the left panel (3) set camera parameters by clicking on Load button in camera parameters section — (4)
    • click button Edit in sync frame section of the left panel (3) and choose a frame with synchronization event — (5)
  • Now videos from all cameras are added and required parameters (camera parameters and synchronization frames) are defined for them.
build-combined-video-workflow.png
  • Evaluate background:
    • using markers (6) and (7) select an interval on the timeline where there is no actor and moving objects (the length of this interval should be about a couple of seconds)
    • alternatively, to set an interval on the timeline you can use controls in section INTERVAL of the toolbar — (8)
    • click Evaluate Background button in the toolbar to evaluate background for all cameras simultaneously from the selected interval on the timeline — (9)
    • alternatively:
      • you can evaluate background for each camera separately using button Edit... in section background in the left panel (3)(A)
      • or you can even load background from file if you've already evaluated it before from different video — (B)
      • for details see below
  • Now everything is ready to be saved into destination .iPiVideo file:
    • using markers (6) and (7) on the timeline or controls (8) in the toolbar, select an interval you want to save to a resulting video
      • for a calibration video it could be an interval where the glowing marker is visible
      • for an action video it could be an interval with needed action sequence or movement
      • or you can save the whole video if you're not sure which exact interval will be used for processing in iPi Mocap Studio
    • click Save Combined Video button in the toolbar and choose destination file name — (C)
    • also before saving you can set compression quality for destination video — (D)
      • but in most cases the default value (70%) works very well providing a good balance between image quality and size of the resulting video
  • Finally you can open the resulting .iPiVideo file in iPi Mocap Studio and process it.

Important Aspects

Now let's discuss some aspects in details:

Cameras and Their Parameters

  1. It is strictly mandatory to add cameras in one and the same order for scene calibration and action videos. In other case results of scene calibration will not match action video(s). By the way, you can use drag-and-drop to reorder cameras in the builder.
  2. Don't forget to apply camera parameters by loading them from appropriate .iPiCam file. Remember that this .iPiCam file must be for this specific camera and for exactly the same resolution and view/FOV settings (FPS may differ). The thing is that camera optics parameters significantly vary from item to item (even when all items are of the same make and model) and depends on resolution and view/FOV settings.
  3. You can adjust the scale of undistort filter right in Combined Video Builder:
build-combined-video-scale.png
Be aware that the same scale value must be used for scene calibration and action videos (per camera, of course). For this reason use this feature with a great care.

Synchronization

To build a combined video, source videos from individual cameras must be synchronized. Current version of Combined Video Builder supports only manual synchronization based on some short/sharp visual event visible in all cameras simultaneously. A frame where such event takes place is called a sync frame. That is, you have to set a sync frame for each source video:

build-combined-video-sync-schema.png

We recommend to use the following visual events for synchronization:

  • the moment when you turn on or off a flash light (or Sony Move)
  • manual firing of an external on-camera flash

For details see How to Record Video with Action Cameras#Synchronization Event.

Example of synchronization using an external on-camera flash in manual mode:

Prev. frame
(before sync)
Sync frameNext frame
(after sync)
cam1 frame-before-sync-cam1.jpgframe-sync-cam1.jpgframe-after-sync-cam1.jpg
cam2 frame-before-sync-cam2.jpgframe-sync-cam2.jpgframe-after-sync-cam2.jpg
cam3 frame-before-sync-cam3.jpgframe-sync-cam3.jpgframe-after-sync-cam3.jpg
cam4 frame-before-sync-cam4.jpgframe-sync-cam4.jpgframe-after-sync-cam4.jpg
(In the 3rd camera the flash is not visible directly, but you can see illumination it produces.)

How to set synchronization frame:

  • Select a camera and click Edit... button in sync frame section of the left panel — see mark (1) on the screenshot below.
  • Additional tab SYNC FRAME opens containing only video from selected camera.
  • Using the timeline (2) choose a frame with synchronization event.
    • it is convenient to use buttons Previous Frame and Next Frame to position precisely on required frame — see marks (3) and (4) on the screenshot below
  • After that click Set to Current Frame button in the toolbar — (5).
  • Now switch back to BUILD tab by clicking on it — (6).
  • Also, you can adjust synchronization frame right in the builder using buttons (7) and (8).
  • Alternatively, you can set a synchronization frame to the current frame shown in the builder using button Current Frame(9).
build-combined-video-how-to-set-sync-frame.jpg

Background

And the last but not the least thing in the way to combine video is background evaluation. Background is an image of clear scene without actors and moving objects:

action-cameras-background-yes-and-no.jpg

There are three ways to evaluate the background:

  1. Simultaneously for all the cameras
    • this method is the easiest and described above in section #Overall Procedure
    • but it requires that clear scene is simultaneously visible in all cameras which can be problematic to achieve in real life conditions
  2. For each camera (source video) separately
    • select required camera and click Edit... button in background section of the left panel — see mark (1) on the screenshot below
    • additional tab BACKGROUND opens with a video from selected camera only
    • using triangles (2) on the timeline or controls (3) in the toolbar, select an interval on the timeline where there is no actor and moving objects (the length of this interval should be about a couple of seconds)
    • after that click Evaluate Background button in the toolbar — (4)
    • optionally you can save the result (evaluated background) to a file on disk to use later with different videos (see below) — (5)
    • now switch back to BUILD tab by clicking on it — (6)
    build-combined-video-how-to-evaluate-background.jpg
  3. Or you can evaluate background based on other video, save it to a file and then just load it when you need the background for appropriate camera.

In any case it is worth to check the background quality after evaluation using Hide Background button in the toolbar of Combined Video Builder:

build-combined-video-hide-background.jpg
(In this particular case background subtraction is not ideal but good enough for motion capture needs.)

Sample

You can use the following sample videos to train in this subject without the need to record your own videos: Sample Videos and Projects (four GoPro).