I Don't Wanna Miss A Thing

My hack is a fairly straightforward way to effect a 360° video based on camera movement. The application here isn’t exactly sophisticated or precise but take it as a jumping off point! The plan is to pause our 360° video when we aren't looking in the direction of our subject (the person turning on and off the camera).

I need a 360° Video!

Chances are you have your own video you'd like to manipulate. If so, please skip ahead.

If you’re still here you need some 360° footage! I grabbed some free raw footage from the Freedom360 website (http://freedom360.us/shop/hero3-black-outdoor-video-sample/).

The video comes from six GoPros configured in Freedom360's housing, meaning we'll have to stitch it together. Once the download is complete, open up Autopano Video Pro.

File ---> Import Videos

Grab all the MP4’s out of the “RawFootage” folder and Import.

Now click on the "Stitch" icon and make sure the settings are for a 6 camera GoPro rig as shown below.

Since this isn't a stitching tutorial, we won't do any fine tuning. The video wont come out perfect but it'll be usable for our purposes.

Press the "Render" icon and set your render Preset to "H.264 HD" and lose the audio track. Feel free to explore larger sizes, I lowered resolution for ease of use. My render settings look as below.

Render out your video. You now have a rectilinear video that looks something like this:

Setting Up Your Sphere

Now that you have your video we need to get it into a viewable state. Create a new Unity 3D project. Drop your video into the project window. Create a new sphere. For ease of use later, I scaled mine to 10 across the x, y, and z.Now to wrap our video around the inside of the sphere we have to start by reversing the normals.

To do that, I grabbed a script for that from the Unity 3D Wiki (http://wiki.unity3d.com/index.php?title=ReverseNormals).

Attach this new script to our sphere.

At this point you could utilize an asset such as "Easy Movie Texture" to apply our 360° video to our sphere. In this project, I created a script using this tutorial:

That code went into the first half of our "VideoController" script and looks like this:

void Awake () {   MovieTexture movie = GetComponent<Renderer>().material.mainTexture as MovieTexture;   movie.loop = true;   movie.Play ();      } 

This script gets attached to our projection sphere as well. Drag the video onto our sphere and it should appear as a shader. Thats it! Next we set up the camera.

Setting Your Camera

Although I never built out the project, I used the camera inside the Google VR SDK to look around my sphere. You can download that here: https://developers.google.com/vr/unity/download

Once you have it, go to Assets --> Import Package --> Custom Package... And import the "GoogleVRForUnity.unitypackage". Under "Prefabs" you should find "GvrViewerMain". Drag that into your Hierarchy. I also went into the "UI" prefabs and attached the "GvrReticle" to my "Main Camera". This gives me a dot at the center of my frame to better visualize the direction of my camera. Hitting play, your project should look something like this:

Detection

Finally, we're going to set up how the camera will detect out hidden triggers.

To do this, we're going to have to create a new C# script that I called "MainCamera". As the name suggests we are attaching it to our Main Camera object.

In this script, we are going to create a Raycast that projects out from our camera and through the center of our image. The code for that looks like:

Ray ray = gameObject.GetComponent<Camera>().ViewportPointToRay(new Vector3(0.5f, .5f, 0)); 

In order to see where the Raycast is in our Scene viewer we're also adding "DrawRay" to illustrate the invisible ray.

Debug.DrawRay(ray.origin, ray.direction * 300, Color.yellow); 

Both of those piece of script will go in Update(), as you can see in the attached script. Since we want to be able to toggle on and off our video, we need to make a public bool that tells us if we are looking at our person. That bool will be determined by whether or not our Raycast hits a specific game object we tag as a "Trigger". All that code put together will give you our "MainCamera" script and it looks like this:

using UnityEngine;
 using System.Collections;
public class MainCamera : MonoBehaviour {
  public bool isLookingAway;
void Update () {
  RaycastHit hit;
  Ray ray = gameObject.GetComponent<Camera>().ViewportPointToRay(new Vector3(0.5f, .5f, 0));
Debug.DrawRay(ray.origin, ray.direction * 300, Color.yellow);
if (Physics.Raycast (ray, out hit) && hit.transform.tag == "Trigger") 
  {
  isLookingAway = false;
  } 
  else 
  {
  isLookingAway = true;
  }
  }
 }

Effecting the Video

Now that we know how to detect a positive Raycast hit on our potential triggers, we need to be able to effect our video. To do that we are going to go back to our "VideoController" script attached to our sphere.

We made "isLookingAway" a public bool for a reason. We need to be able to access it from our "VideoController" script in order to link our RaycastHit to our video. I'm going to create a private bool and call it "check" and link it to "isLookingAway". That piece of code looks like this:

GameObject mainCamera = GameObject.Find("Main Camera");   MainCamera cameraScript = mainCamera.GetComponent<MainCamera> ();   check = cameraScript.isLookingAway;

Lastly, we use our bool "check" in an if statement to pause our movie when we are look away from our subject. The whole script for "VideoController" ends up looking like this:

public class VideoController : MonoBehaviour {
private bool check;
void Awake () {
  MovieTexture movie = GetComponent<Renderer>().material.mainTexture as MovieTexture;
  movie.loop = true;
  movie.Play ();
  
  }
  
  void Update () {
  GameObject mainCamera = GameObject.Find("Main Camera");
  MainCamera cameraScript = mainCamera.GetComponent<MainCamera> ();
  check = cameraScript.isLookingAway;
MovieTexture movie = GetComponent<Renderer>().material.mainTexture as MovieTexture;
if (!check) {
  movie.Play ();
  } else {
  movie.Pause ();
  }
  
  }
 }

Creating our Triggers

We're all set up to pause and play our video based on our triggers now we just need to make them! This is perhaps the least precise piece of this tutorial.

Our triggers can be any game object placed outside your projection sphere, within range of your Raycast, with the tag "Trigger" and placed in the correct position to be hit when looking towards our subject. For that reason this section can be very much up for personal interpretation. I will share how I completed this section but believe there are probably more elegant ways including objects that move with the subject and rounded objects that better wrap around the projection sphere.

My way of placing triggers was simple. I created two cubes. For each, under box collider, I check the "Is Trigger" section and created a new tag called "Trigger" and applied it. Cube inspector should look like this:

Utilizing the game view and the "GvrReticle", I found my edges of view for the subject. By toggling the projection sphere on and off I was able to place and stretch my two cubes so that they covered the full range of movement for our subject.

That way, my ray touched the cube edge right before I'm able to see the subject. My scene view, with cubes and sphere ultimately looked like this:

Conclusion

This very simple method should serve as a jumping off point for anyone looking to create 360° video that is affected by camera movement. Cutting from one scene to another, seamlessly triggering events within a scene, pausing important actions until they're in sight, the possibilities are massive. I'm interested in seeing any improvements on this idea, such as: using timers to activate triggers, moving trigger objects, using multiple raycasts, ect.

Hope this was helpful!