Sahana Rajasekar, Kevin Simons, Clare Lin, Amir Mohtashami

Published April 10, 2014

Go There

Urban 5x5x5x5x15 Project Site selection for Provocation 3: International House corner

Full instructions provided682

Story

Project Introduction

Go There attempts to slow users down when they are in their biggest rush- revealing to them previously-unnoticed aspects of their world and immediate surroundings. As users traverse shortcuts fixed with our installation, Go There attempts to pique their attention by playing different snippets of radio audio. When users are rushing through, our installation plays these snippets at speeds that are too fast for most 'rushers' to discern each individual 'stream' clearly. However, with a slower pace (or by backtracking), users may find their attention caught by a particular station and they (the users) may continue listening to the given station stream by stopping in their tracks altogether.

Go There features computer vision to track users' positions, audio mixing to give a pleasant fading effect between the "switching" of stations, and the interjection of artful static to further associate the site with a sense of the radio. Furthermore, it is our intention that the specific site wherein Go There is installed would determine the type of audio relayed to the user- a library hallway would perhaps play vivid short stories or audiobooks, while a cafe sidewalk may play anything from recipes to breaking news.

Our goal is to provide a more informative and pleasant commute for users while encouraging them to linger on the paths and moments they might have let pass by.

Site Introduction

Our team picked a site located at the corner of International House (a.k.a. "I-House"), finding that it plays into the lives of Berkeley residents in many ways. It was observed that students often walk through the shortcut path when running late for/to class. During other times, some students spend time sitting in the beautiful area, either on the tree branches overlooking the path or on the rocks strewn about it. Lastly, even if they did not directly interact with the site, most students noticed it and had generally positive feelings about the area.

About I-House

Here is the history of I-House provided by the University of California, Berkeley. They also provide information on current events in I-House, information about its residents, and the current goals that I-House is trying to accomplish:

http://ihouse.berkeley.edu/about/history.php

Concept Video

Explore the world, Go There.

The Team at Work

1 / 5 • Amir contemplating files

Hardware Information

Used:

Bose Mini connected via Bluetooth
Webcam (hidden in tree)
Laptop (connected to webcam and running code)

Final Installation Layout

Sketch of hardware layout in situ.

Code

See the included code snippets for detailed reference.

Concerning the audio code used, it:

was written in Python and used a module called Pygame to provide audio mixing
had 4 audio tracks and static constantly playing on a loop
would control the volume level to give the effect of fading, depending on where the person was standing on the path

The vision code:

used Serial input from the webcam to capture the image
used background subtraction, foreground detection, blob recognition through contour mapping, and heuristics to identify moving people
implemented a Kalman filter and defijettering to smooth motion

Computer Vision Code

Download

#!/usr/bin/env python

import os
import cv
import time
import sys
import math
import re
import serial
import pygame
import time
import numpy
from datetime import datetime, timedelta
from ConfigParser import *

CAMERA = 1

class Source:
    def __init__(self, id, flip=True):
        self.capture = cv.CaptureFromCAM(CAMERA)

    def grab_frame(self):
        self.frame = cv.QueryFrame(self.capture)
        if not self.frame:
            print "can't grab frame"
            sys.exit(2)
        cv.Flip(self.frame, None, 1)
        return self.frame

class Music:
    def __init__(self):
        self.songs = []
        self.staticWidth = .2
        self.staticRange = self.staticWidth * 2;

        pygame.init()
        pygame.mixer.init()

        self.staticSoundFile = pygame.mixer.Sound("static.wav")
        self.staticSoundFile.set_volume(0) 
        self.staticSoundFile.play(loops=-1)

        self.addSongObjects("finalChinese.wav")
        self.addSongObjects("finalBrazil.wav")
        self.addSongObjects("finalIndian.wav")
        self.addSongObjects("finalFrance.wav")

        for singleSong in self.songs:
            singleSong.play(loops=-1)

        self.muteAll()

    def setVolumesForPosition(self, position):
        div = 1.0 / (2*len(self.songs))
        #set song volume
        for x in numpy.arange(div, 1, 2*div):
            diff = abs(x - position)
            song = int(x * len(self.songs))
            
            if diff <= div:
                audioLevel = 1 - diff / div
                self.songs[song].set_volume(audioLevel)
            else:
                #audioLevel = 0.5 * self.songs[i].get_volume()
                self.songs[song].set_volume(0)

        #set static volume
        for x in numpy.arange(2*div, 1 - div, 2*div):
            diff = abs(x - position)
            if (diff <= div/2):
                audioLevel = 1 - diff/div
                self.staticSoundFile.set_volume(audioLevel)
                break
            else:
                self.staticSoundFile.set_volume(0)



        # mappedPosition = position * (len(self.songs) - 1)
        # for i in range(len(self.songs)):
        #     difference = abs(i - mappedPosition)
        #     if difference < .5:
        #         audioLevel = 1 - (2 * difference)
        #         self.songs[i].set_volume(audioLevel)
        #     else:
        #         self.songs[i].set_volume(0)
        # # Static controled by global var static width
        # justDecimal = mappedPosition - int(mappedPosition)
        # if justDecimal > (.5 - self.staticWidth) and (justDecimal < .5 +  self.staticWidth):
        #     shiftedDown = justDecimal - ((.5 - self.staticWidth))
        #     staticLevel = 0
        #     if shiftedDown < self.staticWidth: ##going up
        #         staticLevel = shiftedDown/ self.staticWidth      
        #     else: #going down
        #         staticLevel = self.staticRange - shiftedDown
        #         staticLevel = staticLevel/ self.staticWidth      
        #         self.staticSoundFile.set_volume(staticLevel) 
        # else:  
        #     self.staticSoundFile.set_volume(0) 


    def addSongObjects(self, myString):
        soundFile = pygame.mixer.Sound(myString)
        soundFile.set_volume(0)
        self.songs.append(soundFile)

    def muteAll(self):
        for singleSong in self.songs:
            singleSong.set_volume(0)

class Setup:
    def __init__(self):
        self.mixer = None
        self.LoadIniData()
        self.source = Source(self.cameraid)
        self.storage = cv.CreateMemStorage(0)
        self.orig = self.source.grab_frame()

        self.lastDetected = datetime.now()
        self.last = 0
        self.old_center = (0,0)
        self.width = self.orig.width
        self.height = self.orig.height
        self.size = (self.width, self.height)
        self.smallheight = self.working_height
        self.smallwidth = int(self.width * self.smallheight/self.height * 1.0)
        self.smallsize = (self.smallwidth, self.smallheight)

        # alloc mem for images used at various stages
        self.small = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.motion = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.eye = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.mhi = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_32F, 1)
        self.orient = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_32F, 1)
        self.segmask = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_32F, 1)
        self.mask = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_8U, 1)
        self.temp = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.buf = range(10) 
        for i in range(self.n_frames):
            self.buf[i] = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 1)
            cv.SetZero(self.buf[i])

        self.combined = cv.CreateImage((self.smallwidth * self.xwindows, self.smallheight), cv.IPL_DEPTH_8U, 3)

        if self.store:
            self.writer = cv.CreateVideoWriter(self.output, 0, 15, cv.GetSize(self.combined), 1)

        # make window
        #cv.NamedWindow("Motion Detection") 

        #cv.CreateTrackbar('Height', 'Motion Detection', self.height_value, 100, self.change_height)

        #cv.CreateTrackbar('Jitter', 'Motion Detection', self.jitter_value, 100, self.change_jitter)

    def setMixer(self, mixer):
        self.mixer = mixer

    def LoadIniData(self):
        FileName = re.sub(r'\.py$', "", os.path.abspath( __file__ )) + '.ini'
        self.cp=ConfigParser()
        try:
            self.cp.readfp(open(FileName,'r'))
        # f.close()
        except IOError:
            raise Exception,'NoFileError'

        self.store = self.cp.getboolean('Variables', 'store')
        self.cameraid = self.cp.getint('Variables', 'cameraid')
        self.output = self.cp.get('Variables', 'output')
        self.FPS = self.cp.getint('Variables', 'fps')
        self.xwindows = self.cp.getint('Variables', 'xwindows')
        self.working_height = self.cp.getint('Variables', 'working_height')
        self.clocks_per_sec = self.cp.getfloat('Variables', 'clocks_per_sec')
        self.mhi_duration = self.cp.getfloat('Variables', 'mhi_duration')
        self.max_time_delta = self.cp.getfloat('Variables', 'max_time_delta')
        self.min_time_delta = self.cp.getfloat('Variables', 'min_time_delta')
        self.n_frames = self.cp.getint('Variables', 'n_frames')
        self.height_value = self.cp.getint('Variables', 'height_value')
        self.jitter_value = self.cp.getint('Variables', 'jitter_value')

        return

    def process_motion(self,img):
        center = (-1, -1)
        # a lot of stuff from this section was taken from the code motempl.py, 
        #  openCV's python sample code
        timestamp = time.clock() / self.clocks_per_sec # get current time in seconds
        idx1 = self.last
        cv.CvtColor(img, self.buf[self.last], cv.CV_BGR2GRAY) # convert frame to grayscale
        idx2 = (self.last + 1) % self.n_frames 
        self.last = idx2
        silh = self.buf[idx2]
        cv.AbsDiff(self.buf[idx1], self.buf[idx2], silh) # get difference between frames
        cv.Threshold(silh, silh, 30, 1, cv.CV_THRESH_BINARY) # and threshold it
        cv.UpdateMotionHistory(silh, self.mhi, timestamp, self.mhi_duration) # update MHI
        cv.ConvertScale(self.mhi, self.mask, 255./self.mhi_duration, 
                        (self.mhi_duration - timestamp)*255./self.mhi_duration)
        cv.SetZero(img)
        cv.Merge(self.mask, None, None, None, img)
        cv.CalcMotionGradient(self.mhi, self.mask, self.orient, self.max_time_delta, self.min_time_delta, 3)
        seq = cv.SegmentMotion(self.mhi, self.segmask, self.storage, timestamp, self.max_time_delta)
        inc = 0
        a_max = 1000
        max_rect = -1
    
        # there are lots of things moving around
        #  in this case just find find the biggest change on the image
        for (area, value, comp_rect) in seq:
            if comp_rect[2] + comp_rect[3] > 10: # reject small changes
                if area > a_max: 
                    a_max = area
                    max_rect = inc
            inc += 1

        # found it, now just do some processing on the area.
        if max_rect != -1:
            (area, value, comp_rect) = seq[max_rect]
            color = cv.CV_RGB(255, 0,0)
            silh_roi = cv.GetSubRect(silh, comp_rect)
            # calculate number of points within silhouette ROI
            count = cv.Norm(silh_roi, None, cv.CV_L1, None)

            # this rectangle contains the overall motion ROI
            cv.Rectangle(self.motion, (comp_rect[0], comp_rect[1]), 
                         (comp_rect[0] + comp_rect[2], 
                          comp_rect[1] + comp_rect[3]), (0,0,255), 1)

            # the goal is to report back a center of movement contained in a rectangle
            # adjust the height based on the number generated by the slider bar
            h = int(comp_rect[1] + (comp_rect[3] * (float(self.height_value) / 100)))
            # then calculate the center
            center = ((comp_rect[0] + comp_rect[2] / 2), h)

        return center


    def dejitter(self, center):
        # if the center was not found, just draw the center on the old location. 
        if center == (-1,-1):
            color  = cv.CV_RGB(0,225,0)
            r = 15 * .7
            self.old_center = self.old_center
        else: 
            # anti-jittering section
            x1 = self.old_center[0]
            y1 = self.old_center[1]
            x2 = center[0] 
            y2 = center[1]
            dx = x2 - x1
            dy = y2 - y1
            d = math.sqrt( (dx)**2 + (dy)**2 ) # calculate a distance from old location to new.
            angle = math.atan2(dy, dx)  # get an angle while we're doing the math

            if d > self.jitter_value: # if the distance is really far....
                self.old_center = (x2, y2)
                # calculate a near point that exists on the line between the old and new point
                self.old_center = (cv.Round(x1 + self.jitter_value * math.cos(angle)),
                                   cv.Round(y1 + self.jitter_value * math.sin(angle)))
                # and draw it
                r = 15
                color  = cv.CV_RGB(225,0,0)
            else: # the distance is not far so just draw it
                self.old_center = (x2, y2)
                r = 15
                color  = cv.CV_RGB(0,225,0)

        return(r, color)


    def pipeline(self):
        presentation = []
        self.orig = self.source.grab_frame()
        cv.Resize(self.orig, self.small)

        size = self.smallsize

        # store the raw image
        presentation.append((self.small, 'raw'))

        self.motion = cv.CloneImage(self.small)

        # location of the thingy moving on the screen is defined by center
        center = self.process_motion(self.motion)
        
        
        # if an object is found draw the circle on the same location.
        if center != (-1,-1):
            cv.Circle(self.motion, center, cv.Round(15), cv.CV_RGB(0,225,0), 3, cv.CV_AA, 0)
            self.mixer.setVolumesForPosition(center[0] / float(self.smallwidth))
            self.lastDetected = datetime.now()
        else:
        
            if datetime.now() - self.lastDetected > timedelta(seconds=7) and (abs(self.old_center[0] - self.smallwidth/2) > self.smallwidth/4):
                self.mixer.muteAll()
            elif datetime.now() - self.lastDetected > timedelta(minutes=15):
                self.mixer.muteAll()
            else:
                self.mixer.setVolumesForPosition(self.old_center[0] / float(self.smallwidth))

        # store the picture
        presentation.append((self.motion, 'motion')) 

        ####
        # processing of final image
        self.eye = cv.CloneImage(self.small) # get a copy

        (r, color) = self.dejitter(center)
        cv.Circle(self.eye, self.old_center, cv.Round(r), color, 3, cv.CV_AA, 0)

        presentation.append((self.eye, 'final'))

         # combine and show the results
        combined = self.combine_images(presentation)
        cv.ShowImage('Motion detection', combined )

        if self.store:
            cv.WriteFrame(self.writer, self.combined)

    def change_jitter(self, position):
        self.jitter_value = position

    def change_height(self, position):
        self.height_value = position

    def combine_images(self, images):

        font = cv.InitFont(cv.CV_FONT_HERSHEY_SIMPLEX, 1.4, 1.4, 0, 2, 8)

        for i,(image, name) in enumerate(images):
            if image.nChannels == 1:
                cv.Merge(image, image, image, None, self.temp)
            else:
                cv.Copy(image, self.temp)
            xoffset = (i % self.xwindows) * self.smallsize[0]
            yoffset = (i / self.xwindows) * self.smallsize[1]
            cv.SetImageROI(self.combined, (xoffset, yoffset, self.smallsize[0],
                self.smallsize[1]))
            cv.Copy(self.temp, self.combined)
            cv.PutText(self.combined, name, (5, 40), font, (30, 200, 200))
            cv.ResetImageROI(self.combined)
        return self.combined

    def run(self):
        while True:
            t = time.time()
            self.pipeline()
            wait = max(2, (1000/self.FPS)-int((time.time()-t)*1000))
            c = cv.WaitKey(wait) % 0x100
            if c == 27 or c == 10:
                break

if __name__ == '__main__':
    s = Setup()
    mixer = Music()
    s.setMixer(mixer)
    s.run()

World Breakdown

It's a small world, after all!

Looking for France

One of our developers exploring the installation.

User Testing A

One of the first group of users to explore our installation.

User Testing B

Two groups of users exploring the finished installation.

Next Steps?

We foresee many possibilities for the installation we have successfully developed.

A logical next step would be to continue improving the efficacy of our project through fine-tuning the tracking of our technology and improving our code for shadow- and wind-control. We would also like to come up with different options or reactions in tracking multiple people, as well as add to our site with more than one cue (namely the auditory we have currently implemented) in experiencing different countries--for example, having different scents, figurines, or some further interact-able indicators which users can further explore with their different sensory capabilities (olfactory, tactile, temperature, etc.). We would also update our code to include more auditory clips than those we currently possess. Our original idea had been to stream live radio and sound clips from the different countries represented, but this was forgone for simplicity's sake in the final installation. Nonetheless, we would like to better implement this feature in further iterations.

An improved concealment of our hardware would also be an improvement. Early on, our team had considered such accessories as fake rock enclosures (e.g. made of papier mâché) to better hide the laptop (processing software) and the speaker we used to project sounds, as well as the webcamera used to detect user activity, but this was cut due to time constraints in our final project.

The question of adding signage or chalk symbols to better aid people in the discovery of our installation is another issue under consideration.

Music Files

Miscellaneous Project Files

Site Overview

1 / 6 • corner panoramic view

Major Details

1 / 3 • rocks used for sitting

Observations

We noticed that most activity through and around this site occurred on the hour--when students head to class en masse.
We had a difficult time interviewing students who took the shortcut through the site because most of them were intensely rushing to class.
We observed a few people sit on the rock and on the tree. Our pictures of wear-and-use (discarded apple cores, etc.) show that many people enjoy snacking in this site.

Traces of time, wear, and use

1 / 6

Unexpected Things

1 / 5 • random hardware parts

Wi-Fi Networks Present

1 / 3 • Final page of active Wifi signals

Video of I-House Corner A

This video was taken on Wed, 4/9.

Video of I-House Corner B

This video was taken around 11:00am on Thur, 4/10.

Site Interview 1

Site Interview 2

Site Interview 3

Surprises and Challenges

There were a number of confounding variables to our project, many of which had been unforeseen.

The first was that, due to the physical space limitation of our site and path, we had a very small surface area with which to work with. Early on, we decided to limit our span of countries which users could "explore" to only four (4) nations, which four we randomly decided to be China, Brazil, France, and India. This was, understandably, a very narrow window through which users could experience the world, which was compounded by the fact that we defaulted (for simplicity's sake) to choosing only a few excerpts of music for our "radio," excerpts which user testing expressed were strongly associated with the countries we chose. This had the unfortunate result of crafting a rather stereotypical representation of the countries users experienced--something we hope to revisit and ameliorate in future iterations of our installation.

We also had technical complications concerning the vision-/movement-detection. Our installation, which is located outside, generally experiences a fair amount of wind and shadow, which would sometimes interfere with the processing of our technology. We ended up taping a bush which would shake particularly wildly, and made several adjustments to our code in order to better deal with the shadows.

Furthermore, we found that our tiny pathway would often be passed by simply by reason of not being visible enough, located as it is "in the middle of nowhere," and being off the paved road while simultaneously partly obscured by natural elements as rocks, bushes, and dirt. We have considered from user testing the installation of a sign or other indicator to better capture the attention of passerbys, but it can also be argued that news of our installation would surely gather fame in time and could better exist as a special secret of sorts, rewarding those who do risk traveling off the beaten path.

The installation also failed to capture the attention of several users who would be wearing earbuds or headphones as they traversed our path. Granted, it was they who were missing out on experiencing the music of the world!

tracker.py

#!/usr/bin/env python

import os
import cv
import time
import sys
import math
import re
import serial
import pygame
import time
import numpy
from datetime import datetime, timedelta
from ConfigParser import *

CAMERA = 1

class Source:
    def __init__(self, id, flip=True):
        self.capture = cv.CaptureFromCAM(CAMERA)

    def grab_frame(self):
        self.frame = cv.QueryFrame(self.capture)
        if not self.frame:
            print "can't grab frame"
            sys.exit(2)
        cv.Flip(self.frame, None, 1)
        return self.frame

class Music:
    def __init__(self):
        self.songs = []
        self.staticWidth = .2
        self.staticRange = self.staticWidth * 2;

        pygame.init()
        pygame.mixer.init()

        self.staticSoundFile = pygame.mixer.Sound("static.wav")
        self.staticSoundFile.set_volume(0) 
        self.staticSoundFile.play(loops=-1)

        self.addSongObjects("finalChinese.wav")
        self.addSongObjects("finalBrazil.wav")
        self.addSongObjects("finalIndian.wav")
        self.addSongObjects("finalFrance.wav")

        for singleSong in self.songs:
            singleSong.play(loops=-1)

        self.muteAll()

    def setVolumesForPosition(self, position):
        div = 1.0 / (2*len(self.songs))
        #set song volume
        for x in numpy.arange(div, 1, 2*div):
            diff = abs(x - position)
            song = int(x * len(self.songs))
            
            if diff <= div:
                audioLevel = 1 - diff / div
                self.songs[song].set_volume(audioLevel)
            else:
                #audioLevel = 0.5 * self.songs[i].get_volume()
                self.songs[song].set_volume(0)

        #set static volume
        for x in numpy.arange(2*div, 1 - div, 2*div):
            diff = abs(x - position)
            if (diff <= div/2):
                audioLevel = 1 - diff/div
                self.staticSoundFile.set_volume(audioLevel)
                break
            else:
                self.staticSoundFile.set_volume(0)



        # mappedPosition = position * (len(self.songs) - 1)
        # for i in range(len(self.songs)):
        #     difference = abs(i - mappedPosition)
        #     if difference < .5:
        #         audioLevel = 1 - (2 * difference)
        #         self.songs[i].set_volume(audioLevel)
        #     else:
        #         self.songs[i].set_volume(0)
        # # Static controled by global var static width
        # justDecimal = mappedPosition - int(mappedPosition)
        # if justDecimal > (.5 - self.staticWidth) and (justDecimal < .5 +  self.staticWidth):
        #     shiftedDown = justDecimal - ((.5 - self.staticWidth))
        #     staticLevel = 0
        #     if shiftedDown < self.staticWidth: ##going up
        #         staticLevel = shiftedDown/ self.staticWidth      
        #     else: #going down
        #         staticLevel = self.staticRange - shiftedDown
        #         staticLevel = staticLevel/ self.staticWidth      
        #         self.staticSoundFile.set_volume(staticLevel) 
        # else:  
        #     self.staticSoundFile.set_volume(0) 


    def addSongObjects(self, myString):
        soundFile = pygame.mixer.Sound(myString)
        soundFile.set_volume(0)
        self.songs.append(soundFile)

    def muteAll(self):
        for singleSong in self.songs:
            singleSong.set_volume(0)

class Setup:
    def __init__(self):
        self.mixer = None
        self.LoadIniData()
        self.source = Source(self.cameraid)
        self.storage = cv.CreateMemStorage(0)
        self.orig = self.source.grab_frame()

        self.lastDetected = datetime.now()
        self.last = 0
        self.old_center = (0,0)
        self.width = self.orig.width
        self.height = self.orig.height
        self.size = (self.width, self.height)
        self.smallheight = self.working_height
        self.smallwidth = int(self.width * self.smallheight/self.height * 1.0)
        self.smallsize = (self.smallwidth, self.smallheight)

        # alloc mem for images used at various stages
        self.small = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.motion = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.eye = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.mhi = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_32F, 1)
        self.orient = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_32F, 1)
        self.segmask = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_32F, 1)
        self.mask = cv.CreateImage(self.smallsize,cv.IPL_DEPTH_8U, 1)
        self.temp = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 3)
        self.buf = range(10) 
        for i in range(self.n_frames):
            self.buf[i] = cv.CreateImage(self.smallsize, cv.IPL_DEPTH_8U, 1)
            cv.SetZero(self.buf[i])

        self.combined = cv.CreateImage((self.smallwidth * self.xwindows, self.smallheight), cv.IPL_DEPTH_8U, 3)

        if self.store:
            self.writer = cv.CreateVideoWriter(self.output, 0, 15, cv.GetSize(self.combined), 1)

        # make window
        #cv.NamedWindow("Motion Detection") 

        #cv.CreateTrackbar('Height', 'Motion Detection', self.height_value, 100, self.change_height)

        #cv.CreateTrackbar('Jitter', 'Motion Detection', self.jitter_value, 100, self.change_jitter)

    def setMixer(self, mixer):
        self.mixer = mixer

    def LoadIniData(self):
        FileName = re.sub(r'\.py$', "", os.path.abspath( __file__ )) + '.ini'
        self.cp=ConfigParser()
        try:
            self.cp.readfp(open(FileName,'r'))
        # f.close()
        except IOError:
            raise Exception,'NoFileError'

        self.store = self.cp.getboolean('Variables', 'store')
        self.cameraid = self.cp.getint('Variables', 'cameraid')
        self.output = self.cp.get('Variables', 'output')
        self.FPS = self.cp.getint('Variables', 'fps')
        self.xwindows = self.cp.getint('Variables', 'xwindows')
        self.working_height = self.cp.getint('Variables', 'working_height')
        self.clocks_per_sec = self.cp.getfloat('Variables', 'clocks_per_sec')
        self.mhi_duration = self.cp.getfloat('Variables', 'mhi_duration')
        self.max_time_delta = self.cp.getfloat('Variables', 'max_time_delta')
        self.min_time_delta = self.cp.getfloat('Variables', 'min_time_delta')
        self.n_frames = self.cp.getint('Variables', 'n_frames')
        self.height_value = self.cp.getint('Variables', 'height_value')
        self.jitter_value = self.cp.getint('Variables', 'jitter_value')

        return

    def process_motion(self,img):
        center = (-1, -1)
        # a lot of stuff from this section was taken from the code motempl.py, 
        #  openCV's python sample code
        timestamp = time.clock() / self.clocks_per_sec # get current time in seconds
        idx1 = self.last
        cv.CvtColor(img, self.buf[self.last], cv.CV_BGR2GRAY) # convert frame to grayscale
        idx2 = (self.last + 1) % self.n_frames 
        self.last = idx2
        silh = self.buf[idx2]
        cv.AbsDiff(self.buf[idx1], self.buf[idx2], silh) # get difference between frames
        cv.Threshold(silh, silh, 30, 1, cv.CV_THRESH_BINARY) # and threshold it
        cv.UpdateMotionHistory(silh, self.mhi, timestamp, self.mhi_duration) # update MHI
        cv.ConvertScale(self.mhi, self.mask, 255./self.mhi_duration, 
                        (self.mhi_duration - timestamp)*255./self.mhi_duration)
        cv.SetZero(img)
        cv.Merge(self.mask, None, None, None, img)
        cv.CalcMotionGradient(self.mhi, self.mask, self.orient, self.max_time_delta, self.min_time_delta, 3)
        seq = cv.SegmentMotion(self.mhi, self.segmask, self.storage, timestamp, self.max_time_delta)
        inc = 0
        a_max = 1000
        max_rect = -1
    
        # there are lots of things moving around
        #  in this case just find find the biggest change on the image
        for (area, value, comp_rect) in seq:
            if comp_rect[2] + comp_rect[3] > 10: # reject small changes
                if area > a_max: 
                    a_max = area
                    max_rect = inc
            inc += 1

        # found it, now just do some processing on the area.
        if max_rect != -1:
            (area, value, comp_rect) = seq[max_rect]
            color = cv.CV_RGB(255, 0,0)
            silh_roi = cv.GetSubRect(silh, comp_rect)
            # calculate number of points within silhouette ROI
            count = cv.Norm(silh_roi, None, cv.CV_L1, None)

            # this rectangle contains the overall motion ROI
            cv.Rectangle(self.motion, (comp_rect[0], comp_rect[1]), 
                         (comp_rect[0] + comp_rect[2], 
                          comp_rect[1] + comp_rect[3]), (0,0,255), 1)

            # the goal is to report back a center of movement contained in a rectangle
            # adjust the height based on the number generated by the slider bar
            h = int(comp_rect[1] + (comp_rect[3] * (float(self.height_value) / 100)))
            # then calculate the center
            center = ((comp_rect[0] + comp_rect[2] / 2), h)

        return center


    def dejitter(self, center):
        # if the center was not found, just draw the center on the old location. 
        if center == (-1,-1):
            color  = cv.CV_RGB(0,225,0)
            r = 15 * .7
            self.old_center = self.old_center
        else: 
            # anti-jittering section
            x1 = self.old_center[0]
            y1 = self.old_center[1]
            x2 = center[0] 
            y2 = center[1]
            dx = x2 - x1
            dy = y2 - y1
            d = math.sqrt( (dx)**2 + (dy)**2 ) # calculate a distance from old location to new.
            angle = math.atan2(dy, dx)  # get an angle while we're doing the math

            if d > self.jitter_value: # if the distance is really far....
                self.old_center = (x2, y2)
                # calculate a near point that exists on the line between the old and new point
                self.old_center = (cv.Round(x1 + self.jitter_value * math.cos(angle)),
                                   cv.Round(y1 + self.jitter_value * math.sin(angle)))
                # and draw it
                r = 15
                color  = cv.CV_RGB(225,0,0)
            else: # the distance is not far so just draw it
                self.old_center = (x2, y2)
                r = 15
                color  = cv.CV_RGB(0,225,0)

        return(r, color)


    def pipeline(self):
        presentation = []
        self.orig = self.source.grab_frame()
        cv.Resize(self.orig, self.small)

        size = self.smallsize

        # store the raw image
        presentation.append((self.small, 'raw'))

        self.motion = cv.CloneImage(self.small)

        # location of the thingy moving on the screen is defined by center
        center = self.process_motion(self.motion)
        
        
        # if an object is found draw the circle on the same location.
        if center != (-1,-1):
            cv.Circle(self.motion, center, cv.Round(15), cv.CV_RGB(0,225,0), 3, cv.CV_AA, 0)
            self.mixer.setVolumesForPosition(center[0] / float(self.smallwidth))
            self.lastDetected = datetime.now()
        else:
        
            if datetime.now() - self.lastDetected > timedelta(seconds=7) and (abs(self.old_center[0] - self.smallwidth/2) > self.smallwidth/4):
                self.mixer.muteAll()
            elif datetime.now() - self.lastDetected > timedelta(minutes=15):
                self.mixer.muteAll()
            else:
                self.mixer.setVolumesForPosition(self.old_center[0] / float(self.smallwidth))

        # store the picture
        presentation.append((self.motion, 'motion')) 

        ####
        # processing of final image
        self.eye = cv.CloneImage(self.small) # get a copy

        (r, color) = self.dejitter(center)
        cv.Circle(self.eye, self.old_center, cv.Round(r), color, 3, cv.CV_AA, 0)

        presentation.append((self.eye, 'final'))

         # combine and show the results
        combined = self.combine_images(presentation)
        cv.ShowImage('Motion detection', combined )

        if self.store:
            cv.WriteFrame(self.writer, self.combined)

    def change_jitter(self, position):
        self.jitter_value = position

    def change_height(self, position):
        self.height_value = position

    def combine_images(self, images):

        font = cv.InitFont(cv.CV_FONT_HERSHEY_SIMPLEX, 1.4, 1.4, 0, 2, 8)

        for i,(image, name) in enumerate(images):
            if image.nChannels == 1:
                cv.Merge(image, image, image, None, self.temp)
            else:
                cv.Copy(image, self.temp)
            xoffset = (i % self.xwindows) * self.smallsize[0]
            yoffset = (i / self.xwindows) * self.smallsize[1]
            cv.SetImageROI(self.combined, (xoffset, yoffset, self.smallsize[0],
                self.smallsize[1]))
            cv.Copy(self.temp, self.combined)
            cv.PutText(self.combined, name, (5, 40), font, (30, 200, 200))
            cv.ResetImageROI(self.combined)
        return self.combined

    def run(self):
        while True:
            t = time.time()
            self.pipeline()
            wait = max(2, (1000/self.FPS)-int((time.time()-t)*1000))
            c = cv.WaitKey(wait) % 0x100
            if c == 27 or c == 10:
                break

if __name__ == '__main__':
    s = Setup()
    mixer = Music()
    s.setMixer(mixer)
    s.run()

Embed the widget on your own site

Go There

Go There

Story

Project Introduction

Site Introduction

About I-House

Concept Video

The Team at Work

Hardware Information

Final Installation Layout

Code

Computer Vision Code

Computer Vision Code

World Breakdown

Looking for France

User Testing A

User Testing B

Next Steps?

Music Files

finalBrazil.m4a

finalIndian.m4a

finalChinese.m4a

static.wav

Miscellaneous Project Files

Copy%20of%20Presentation%20-%20Studio.pptx

Site Overview

Major Details

Observations

Traces of time, wear, and use

Unexpected Things

Wi-Fi Networks Present

Video of I-House Corner A

Video of I-House Corner B

Site Interview 1

Site Interview 2

Site Interview 3

Surprises and Challenges

Code

tracker.py

Credits

Sahana Rajasekar

Kevin Simons

Clare Lin

Amir Mohtashami

Comments

Related channels and tags