Playback

Browser playback

PC Browser playback

In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select Web mode (which is the default)

Translate (x,y) position: click left mouse button and drag
Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
Reset position: double click
Rotate angle: hold down center mouse button (scroll wheel) and drag
Step-in/out: mouse scroll wheel up/down

Phone Browser playback

In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select Web mode (which is the default)

Translate (x,y) position: two finger swipe
Arc motion (x,y,z) position & orientation: one finger swipe
Step-in/out: pinch
Reset position: double tap

App playback

Windows PC app playback

Install the Windows PC 3D video player app
In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select 3D to launch the Windows player app

Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
Reset position: double click
Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
Modify navigation sensitivity: select settings , adjust sensitivity slider
Modify focal point: select settings , adjust focus slider
Toggle GUI on or off: T keyboard key
Step-in/out: mouse scroll wheel up/down

Android app playback

Install the 3D video player app from the Google Playstore
In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select 3D to launch the Windows player app
Translate (x,y) position: two finger swipe
Arc motion (x,y,z) position & orientation: one finger swipe
Step-in/out: pinch
Reset position: double tap

iPhone Capture

Overview

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iad Pro, or with theKinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor.

Captured videos arsubmitted to the Vimmerse platform for content preparation.

Creators can also create "bullet videos" in which the creator controls the 6DoF navigation and outputs a normal 2D video with 3D effects that is playable on any video player.

Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a masked video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video.

1. Capture with the Vimmerse Capture app

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iPad Pro, or with the Kinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. Captured videos are for content preparation.

Install the Vimmerse Capture app from the Apple App Store.

Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.

Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
Record a video by hitting the record button and stop button.

Select between choosing to capture a background or to start recording a video

Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video

Hit the record button to start creating capturing a video

Optionally manually set the depth range in the Settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.

Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m)

2. Upload your content

Optionally select the editor icon to trim the video.

Select the Gallery tab to preview your captured video as side-by-side regular video and depth

Optionally rename the capture and/or add a description by selecting the editing pencil and entering text

Select the cloud icon to upload the video to the Vimmerse website.
If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.

3. Masking videos

To mask your video on a computer, first download your captured content to a computer. Visit your user content page to access your captured video.

Select desired video

Click the Download button and select “Download source views”. Example content can be found here.

Masking with RunwayML

Use After Effects or RunwayML’s green screen tool (https://runwayml.com/green-screen/) or After Effects’ rotobrush features (https://helpx.adobe.com/after-effects/using/roto-brush-refine-matte.html) or other tool to rotoscope the texture source video (which is named v0_t.mp4), to create a alpha mask, of moving foreground objects and people. The iPhone captured texture video is 720x960 resolution, so use that resolution.
If using RunwayML’s green screen tool, select “Export Matte” to export the alpha mask.

Masking with After Effects

Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline.
Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.

Create a new solid layer and set its color to white.
Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.

4. Background Generation

If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.

For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.

Alternatively, any video can be used as a background video, which may optionally contain depth.

You can create the depth for a background video layer with Runway’s MiDaS model or with the DepthScanner plugin for After Effects (https://aescripts.com/depth-scanner/).

5. Re-Uploading

The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.

Masked video: v0_m.mp4
Texture video: v0_t.mp4
Metadata file: any name with a .json extension
(Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
(Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
(Optional) audio file: any name with .aac or .mp3 extension
Depth/Geometry video: v0_d.mp4

On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
After the files have been uploaded, select “Submit for Processing”.
You will be notified by email after the processing is complete or if there was an error.

Kinect DK Capture Using Depthkit

Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv‍

1. Recording

Use DepthKit to record a scene, following the instructions in the DepthKit documentation at https://docs.depthkit.tv/docs

2. Export

Select the EDIT tab
Select REFINE
Select Enable refinement
Set the Filter Size and Fill Amount to the maximum values
Select ISOLATE
Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
Select EXPORT
Select Combined Per Pixel video
Export

3. Upload

Login to the Vimmerse platform in a browser, select Upload content.
In the Capture format pulldown, select Auto, which is the default.
Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
On the Upload content page, upload the .mp4 and .txt files from your capture.
After uploading is complete, select Submit for Processing.

Create bullet videos

Bullet videos are 2D videos with creator-controlled 3D navigation, which can be posted to 3rd party video sharing services, including social media.

The bullet video creation tool app is available for Windows.

Install the bullet video creation tool, Vimmerse Creator
Test by downloading some of the 3D videos on the featured content page.
Command line instructions
Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
In the command prompt, launch the Vimmerse Creator with following command line:
VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
or
VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
App interface instructions
Launch the Vimmerse Creator app from the Windows menu
Select the file icon to open the input 3D video file

Create navigation path:
Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (+) icon to set a key frame. Resume playback and repeat to create additional key frames.

Key frame locations will be indicated on the timeline on the bottom. To remove a key frame, select it and click on the delete icon (X).

Optionally create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame icon (+) to set a key frame. Repeat with different duration values and different poses.

When finished, click the export icon. If you used the command line, the files names you specified will be used, e.g .the output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv. If the app launcher was used, the output bullet video and pose trace files will be given default names and placed in that same directory as the 3D video file..

Playback

Browser playback

PC Browser playback

In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select Web mode (which is the default)

Translate (x,y) position: click left mouse button and drag
Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
Reset position: double click
Rotate angle: hold down center mouse button (scroll wheel) and drag
Step-in/out: mouse scroll wheel up/down

Phone Browser playback

In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select Web mode (which is the default)

Translate (x,y) position: two finger swipe
Arc motion (x,y,z) position & orientation: one finger swipe
Step-in/out: pinch
Reset position: double tap

App playback

Windows PC app playback

Install the Windows PC 3D video player app
In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select 3D to launch the Windows player app

Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
Step-in/out: mouse scroll wheel up/down
Reset position: double click
Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
Modify navigation sensitivity: select settings , adjust sensitivity slider
Modify focal point: select settings , adjust focus slider
Toggle GUI on or off: T keyboard key

Android app playback

Install the 3D video player app from the Google Playstore
In a browser, locate a 3D video on the Vimmerse website. View featured content.
Select 3D to launch the Windows player app
Translate (x,y) position: two finger swipe
Arc motion (x,y,z) position & orientation: one finger swipe
Step-in/out: pinch
Reset position: double tap

iPhone Capture

Overview

iPhone Capture

1. Capture with the Vimmerse Capture app

Install the Vimmerse Capture app from the Apple App Store.

Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.

Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
Record a video by hitting the record button and stop button.

Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.

iPhone Capture

2. Upload your content

Optionally select the editor icon to trim the video.

Select the Gallery tab to preview your captured video as side-by-side regular video and depth

Optionally rename the capture and/or add a description by selecting the editing pencil and entering text

Select the cloud icon to upload the video to the Vimmerse website.
If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.

iPhone Capture

3. Masking videos

To mask your video on a computer, first download your captured content to a computer. Visit your user content page to access your captured video.

Select desired video

Click the Download button and select “Download source views”. Example content can be found here.

Masking with RunwayML

Use After Effects or RunwayML’s green screen tool (https://runwayml.com/green-screen/) or After Effects’ rotobrush features (https://helpx.adobe.com/after-effects/using/roto-brush-refine-matte.html) or other tool to rotoscope the texture source video (which is named v0_t.mp4), to create a alpha mask, of moving foreground objects and people. The iPhone captured texture video is 720x960 resolution, so use that resolution.
If using RunwayML’s green screen tool, select “Export Matte” to export the alpha mask.

Masking with After Effects

Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline.
Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.

Create a new solid layer and set its color to white.
Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.

iPhone Capture

4. Background Generation

Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a maksing video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video.

If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.

For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.

Alternatively, any video can be used as a background video, which may optionally contain depth.

You can create the depth for a background video layer with Runway’s MiDaS model or with the DepthScanner plugin for After Effects (https://aescripts.com/depth-scanner/).

iPhone Capture

5. Re-Uploading

The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.

Masked video: v0_m.mp4
Texture video: v0_t.mp4
Metadata file: any name with a .json extension
(Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
(Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
(Optional) audio file: any name with .aac or .mp3 extension
Depth/Geometry video: v0_d.mp4

On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
After the files have been uploaded, select “Submit for Processing”.
You will be notified by email after the processing is complete or if there was an error.

Kinect DK Capture Using Depthkit

Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv‍

1. Recording

Use DepthKit to record a scene, following the instructions in the DepthKit documentation at https://docs.depthkit.tv/docs

2. Export

Select the EDIT tab
Select REFINE
Select Enable refinement
Set the Filter Size and Fill Amount to the maximum values
Select ISOLATE
Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
Select EXPORT
Select Combined Per Pixel video
Export

3. Upload

Login to the Vimmerse platform in a browser, select Upload content.
In the Capture format pulldown, select Auto, which is the default.
Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
On the Upload content page, upload the .mp4 and .txt files from your capture.
After uploading is complete, select Submit for Processing.

Create bullet videos

Install the bullet video creation tool, Vimmerse Creator
Test by downloading some of the 3D videos on the featured content page.
Command line instructions
Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
In the command prompt, launch the Vimmerse Creator with following command line:
VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
or
VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
App interface instructions
Launch the Vimmerse Creator app from the Windows menu
Select the file icon to open the input 3D video file

Create navigation path:
Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (+) icon to set a key frame. Resume playback and repeat to create additional key frames.

Key frame locations will be indicated on the timeline on the bottom. To remove a key frame, select it and click on the delete icon (X).

Optionally create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame icon (+) to set a key frame. Repeat with different duration values and different poses.

When finished, click the export icon. If you used the command line, the files names you specified will be used, e.g .the output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv. If the app launcher was used, the output bullet video and pose trace files will begiven default names and placed in that same directory as the 3D video file..