Visualize MENTAL STATE in virtual reality

I would link a video to this but since I do not have a muse in my possession anymore, I'm unable to show a demo. Basically, if you're relaxed, the environment will change colors (blue) to match your mood. In a better scenario, the virtual world would shapeshift to turn into some relaxed environment such as a beach or quiet room. If you were energetic, the environment would turn red. However, you can download our Github files and try out the demo for yourself if you have an Oculus and Muse!

My team and I won CalTech's engineering competition for best medical hack using this project. https://www.caltech.edu/news/caltech-hosts-coding-competition-high-school-and-college-students-50039

Inspiration

We were looking at the cool hardware available at HackTech and thought of ways to combine them to create novel solutions. One idea that stuck out was mixing Muse and Oculus to do neuro-feedback in a virtual reality environment.

What it does

ThinkVR tracks brain waves with Muse and passes it to Unity which creates a dynamic environment based on the data. In this environment, changes in the user's mental Estate are reflected in a immersive virtual reality environment. The measurements Muse takes are harmless to people and only provide information about your general state. The muse gives us 4 channel EEG recordings so determining what each EEG wave corresponds which each emotion has helped us significantly. Brain sensing has been used in hospitals and research institutions for over a century to study various aspects of the brain's function. Neuro-feedback is already backed by research showing that it's an effective treatment for mental health and this approaches it from a new angle. Also you can order food using the PostMates API.

How I built it

Muse API on back end to pass data to Unity, with various API added to provide additional functionality.

Challenges I ran into

I wasn't familiar with Unity or C# so understanding a 3 dimensional layout based in a completely new software proved especially challenging. Hooking up and interpreting muse data in an Oculus app also proved quite challenging.

Accomplishments that I'm proud of

In Unity, you can ray-cast crosshairs to find where the user is looking. Another accomplishment was that Muse sends data through TCP but C# only has a UDP listener so we hacked Muse to send data in that format instead. Users can order food within the VR app.

What I learned

The basics of Unity, the world of Muse. There are a lot!

What's next for Think VR

More moods and environments, custom hardware.