Instructions

Learn how to use Vimmerse's 3D immersive video technology.

All
Playback
iPhone
Capture
Uploading
Masking
Backgrounds
Re-Uploading
Kinect DK
Bullet Videos

Playback

Browser playback

PC Browser playback

  • In a browser, locate a 3D video on the Vimmerse website. View featured content.
  • Select Web mode (which is the default)
  • Translate (x,y) position: click left mouse button and drag
  • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
  • Reset position: double click
  • Rotate angle: hold down center mouse button (scroll wheel) and drag
  • Step-in/out: mouse scroll wheel up/down

Phone Browser playback

  • In a browser, locate a 3D video on the Vimmerse website. View featured content.
  • Select Web mode (which is the default)
  • Translate (x,y) position: two finger swipe
  • Arc motion (x,y,z) position & orientation: one finger swipe
  • Step-in/out: pinch
  • Reset position: double tap

App playback

Windows PC app playback

  • Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
  • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
  • Reset position: double click
  • Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
  • Modify navigation sensitivity: select settings , adjust sensitivity slider
  • Modify focal point: select settings , adjust focus slider
  • Toggle GUI on or off: T keyboard key
  • Step-in/out: mouse scroll wheel up/down

Android app playback

iPhone Capture

Overview

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iad Pro, or with theKinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. 

Captured videos arsubmitted to the Vimmerse platform for content preparation. 

Creators can also create "bullet videos" in which the creator controls the 6DoF navigation and outputs a normal 2D video with 3D effects that is playable on any video player. 

Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a masked video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video. 

1. Capture with the Vimmerse Capture app

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iPad Pro, or with the Kinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. Captured videos are for content preparation.

  • Install the Vimmerse Capture app from the Apple App Store.
Download the iOS app on the app store
  • Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
  • If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
  • Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.
  • Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
  • Record a video by hitting the record button and stop button.
Select between choosing to capture a background or to start recording a videoOptionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) videoHit the record button to start creating capturing a video
  • Optionally manually set the depth range in the Settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.
Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m)

2. Upload your content

    • Optionally select the editor icon to trim the video.
    • Select the Gallery tab to preview your captured video as side-by-side regular video and depth
    • Optionally rename the capture and/or add a description by selecting the editing pencil and entering text
    • Select the cloud icon to upload the video to the Vimmerse website.
    • If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
    • Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.

    3. Masking videos

    • To mask your video on a computer, first download your captured content to a computer. Visit your user content page to access your captured video.
    User content page screenshot
    • Select desired video
    My Content page example
    • Click the Download button and select “Download source views”. Example content can be found here.
    User content page screenshot

    Masking with RunwayML

    Screenshot from Runway ML

        Masking with After Effects

        • Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline. 
        • Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.
        After Effects Screenshot
        • Create a new solid layer and set its color to white.
        • Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.
        After effects with Rotobrushing example

          4. Background Generation

          • If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
          • Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.
          • For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
          • For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.
          • Alternatively, any video can be used as a background video, which may optionally contain depth.

          5. Re-Uploading

          • The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.
          • Masked video: v0_m.mp4
          • Texture video: v0_t.mp4
          • Metadata file: any name with a .json extension
          • (Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
          • (Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
          • (Optional) audio file: any name with .aac or .mp3 extension
          • Depth/Geometry video: v0_d.mp4
          • On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
          • Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
          • After the files have been uploaded, select “Submit for Processing”.
          • You will be notified by email after the processing is complete or if there was an error.

          Kinect DK Capture Using Depthkit

          Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv

          1. Recording

          2. Export

          • Select the EDIT tab
          • Select REFINE
          • Select Enable refinement
          • Set the Filter Size and Fill Amount to the maximum values
          • Select ISOLATE
          • Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
          • Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
          • Select EXPORT
          • Select Combined Per Pixel video
          • Export
          Preferred Depthkit Export Settings

          3. Upload

          • Login to the Vimmerse platform in a browser, select Upload content.
          • In the Capture format pulldown, select Auto, which is the default.
          • Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
          • On the Upload content page, upload the .mp4 and .txt files from your capture.
          • After uploading is complete, select Submit for Processing.

          Create bullet videos

          Bullet videos are 2D videos with creator-controlled 3D navigation, which can be posted to 3rd party video sharing services, including social media.

          The bullet video creation tool app is available for Windows.

          • Install the bullet video creation tool, Vimmerse Creator
          • Test by downloading some of the 3D videos on the featured content page.
          • Command line instructions
          • Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
          • Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
          • In the command prompt, launch the Vimmerse Creator with following command line:
            VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
            or
            VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
          • App interface instructions
          • Launch the Vimmerse Creator app from the Windows menu
          • Select the file icon to open the input 3D video file
          • Create navigation path:
          • Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (+) icon to set a key frame. Resume playback and repeat to create additional key frames.
          • Key frame locations will be indicated on the timeline on the bottom. To remove a key frame, select it and click on the delete icon (X).
          • Optionally create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame icon (+) to set a key frame. Repeat with different duration values and different poses.
              • When finished, click the export icon. If you used the command line, the files names you specified will be used, e.g .the output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv. If the app launcher was used, the output bullet video and pose trace files will be given default names and placed in that same directory as the 3D video file..

              Playback

              Browser playback

              PC Browser playback

              • In a browser, locate a 3D video on the Vimmerse website. View featured content.
              • Select Web mode (which is the default)
              • Translate (x,y) position: click left mouse button and drag
              • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
              • Reset position: double click
              • Rotate angle: hold down center mouse button (scroll wheel) and drag
              • Step-in/out: mouse scroll wheel up/down

              Phone Browser playback

              • In a browser, locate a 3D video on the Vimmerse website. View featured content.
              • Select Web mode (which is the default)
              • Translate (x,y) position: two finger swipe
              • Arc motion (x,y,z) position & orientation: one finger swipe
              • Step-in/out: pinch
              • Reset position: double tap

              App playback

              Windows PC app playback

              • Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
              • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
              • Step-in/out: mouse scroll wheel up/down
              • Reset position: double click
              • Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
              • Modify navigation sensitivity: select settings , adjust sensitivity slider
              • Modify focal point: select settings , adjust focus slider
              • Toggle GUI on or off: T keyboard key

              Android app playback

              iPhone Capture

              Overview

              3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iad Pro, or with theKinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. 

              Captured videos arsubmitted to the Vimmerse platform for content preparation. 

              Creators can also create "bullet videos" in which the creator controls the 6DoF navigation and outputs a normal 2D video with 3D effects that is playable on any video player. 

              Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a masked video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video. 

              iPhone Capture

              1. Capture with the Vimmerse Capture app

              • Install the Vimmerse Capture app from the Apple App Store.
              Download the iOS app on the app store
              • Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
              • If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
              • Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.
              • Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
              • Record a video by hitting the record button and stop button.
              Select between choosing to capture a background or to start recording a videoOptionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) videoHit the record button to start creating capturing a video
              Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m)
              • Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.

              iPhone Capture

              2. Upload your content

                • Optionally select the editor icon to trim the video.
                • Select the Gallery tab to preview your captured video as side-by-side regular video and depth
                • Optionally rename the capture and/or add a description by selecting the editing pencil and entering text
                • Select the cloud icon to upload the video to the Vimmerse website.
                • If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
                • Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.

                iPhone Capture

                3. Masking videos

                • To mask your video on a computer, first download your captured content to a computer. Visit your user content page to access your captured video.
                User content page screenshot
                • Select desired video
                My Content page example
                • Click the Download button and select “Download source views”. Example content can be found here.
                User content page screenshot

                Masking with RunwayML

                Screenshot from Runway ML

                    Masking with After Effects

                    • Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline. 
                    • Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.
                    After Effects Screenshot
                    • Create a new solid layer and set its color to white.
                    • Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.
                    After effects with Rotobrushing example

                      iPhone Capture

                      4. Background Generation

                      Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a maksing video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video.

                      • If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
                      • Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.
                      • For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
                      • For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.
                      • Alternatively, any video can be used as a background video, which may optionally contain depth.

                      iPhone Capture

                      5. Re-Uploading

                      • The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.
                      • Masked video: v0_m.mp4
                      • Texture video: v0_t.mp4
                      • Metadata file: any name with a .json extension
                      • (Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
                      • (Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
                      • (Optional) audio file: any name with .aac or .mp3 extension
                      • Depth/Geometry video: v0_d.mp4
                      • On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
                      • Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
                      • After the files have been uploaded, select “Submit for Processing”.
                      • You will be notified by email after the processing is complete or if there was an error.

                      Kinect DK Capture Using Depthkit

                      Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv

                      1. Recording

                      2. Export

                      • Select the EDIT tab
                      • Select REFINE
                      • Select Enable refinement
                      • Set the Filter Size and Fill Amount to the maximum values
                      • Select ISOLATE
                      • Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
                      • Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
                      • Select EXPORT
                      • Select Combined Per Pixel video
                      • Export
                      Preferred Depthkit Export Settings

                      3. Upload

                      • Login to the Vimmerse platform in a browser, select Upload content.
                      • In the Capture format pulldown, select Auto, which is the default.
                      • Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
                      • On the Upload content page, upload the .mp4 and .txt files from your capture.
                      • After uploading is complete, select Submit for Processing.

                      Create bullet videos

                      Bullet videos are 2D videos with creator-controlled 3D navigation, which can be posted to 3rd party video sharing services, including social media.

                      The bullet video creation tool app is available for Windows.

                      • Install the bullet video creation tool, Vimmerse Creator
                      • Test by downloading some of the 3D videos on the featured content page.
                      • Command line instructions
                      • Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
                      • Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
                      • In the command prompt, launch the Vimmerse Creator with following command line:
                        VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
                        or
                        VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
                      • App interface instructions
                      • Launch the Vimmerse Creator app from the Windows menu
                      • Select the file icon to open the input 3D video file
                      • Create navigation path:
                      • Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (+) icon to set a key frame. Resume playback and repeat to create additional key frames.
                      • Key frame locations will be indicated on the timeline on the bottom. To remove a key frame, select it and click on the delete icon (X).
                      • Optionally create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame icon (+) to set a key frame. Repeat with different duration values and different poses.
                          • When finished, click the export icon. If you used the command line, the files names you specified will be used, e.g .the output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv. If the app launcher was used, the output bullet video and pose trace files will begiven default names and placed in that same directory as the 3D video file..