Handsfree Operation Theater Lamp for Surgeons

About

This project is about building a lighting system for Operation Theater table which allows surgeons to adjust light with voice command. Existing solution requires manual adjustment of lamp, this solution will allow hands-free adjustment of lamp

Instead of manually adjusting lighting angle and intensity in OT, adjust the lamps with voice command from Surgeons.

Hardware

Following Hardware will be required

• Raspberry Pi 3A+ - running Raspbian Stretch and Matrix OS
• MATRIXCreator - for built-in microphone
• Micro-USB power bank 2A, 5 Volts
• Micro SD Card for burning Raspbian Stretch OS
• Micro-USB Cable
• Network connection -WiFi
• Stepper Motor - movement of light
• MOSFET - controlling brightness
• LED Array - Lighting
• Nylon Gears - Mechanical movement

Required Software

• Node.js: Dependancy for Sam CLI Tool.
• Etcher.io: To easily flash our Raspbian Stretch image.
• Snips' Sam CLI Tool: Creates & manages Snip assistants on Raspberry Pi.

A registered account on snips.ai

Step 0: Preparing Pi, Matrix and Snips

First, Matrix creator is set up according to following official tutorial

• MATRIX Creator Setup

Once setup is complete SSH is enabled on Raspberry Pi. This allows the SAM CLI tool to connect to Pi.

Next Matrix Lite and Snips is configured according to following tutorials

• Install MATRIX Lite JS
• Configure Snips.ai

Step 1: Connecting SAM To Raspberry Pi

From computer's command terminal, sign in through the Sam CLI Tool.

sam login

Then connect to Raspberry Pi. When prompted for a username & password.

sam connect 192.168.10.3

Check if Snips is running.

sam status
# Example output
# If all services are not (running), use the `sam init` command
Service status:
snips-analytics .............. 0.64.0 (running)
snips-asr .................... 0.64.0 (running)
snips-audio-server ........... 0.64.0 (running)
snips-dialogue ............... 0.64.0 (running)
snips-hotword ................ 0.64.0 (running)
snips-nlu .................... 0.64.0 (running)
snips-skill-server ........... 0.64.0 (running)
snips-tts .................... 0.64.0 (running)

Begin viewing the Snips event stream. This will let us verify if the intents are being heard.

sam watch

Step 2: Creating A Snips Assistant & App

Next, from a signed in Snips.ai account, an assiatant is created. An appropriate wakeword is chosen. For example, "Light" is the wakeword.

Step 3: Adding an Intent to the App

Intents enables Snips to handle user requests. For this Assistant, we'll start by creating an intent for

• Turning On/Off the External LEDs.
• Adjust Brightness of the LEDs with PWM
• Move LEDs with Stepper Motor

Assigning a new intent called lightState for lights app.

Once the intent is defined, user needs to extract the key words like on/off, increase/decrease brightness, move left/right etc. This is where slots come in.

A new slot called power is made and a custom slot type with the same name.

power slot type has on & off as values.

The example sentences contains the words on/off, increase brightness/decrease brightness and move left/right/up/down options.

These words are highlighted in the example sentences to assign them to recently created power slot.

Step 4: Deploying an Assistant & Catching Intents

Now the assistant can be deployed. On the bottom right of the page will be a Deploy Assistant button.

By using the Sam CLI Tool to deploy the assistant on Raspberry Pi. here is an example of the command to use.

Step 5: Creating assistant.js (Raspberry Pi)

The following step is to setup a MATRIX Lite project with Snips. assistant.js that will be used to listen and respond to events from the Snips assistant

Following files are copied the 3 required JavaScript files. to cd /home/pi/lite_js directory

MQTT needs to be be installed and used to listen in on events from Snips.

npm install mqtt --save

PwmLED.js combines all the code from stepper.js & mosfet.js to control the LED brightness, movement and on/off switch through voice. It will listen to lightState intents and read slots accordingly.

var mqtt = require('mqtt');
var client  = mqtt.connect('mqtt://localhost', { port: 1883 });
var mosfet = require('./mosfet.js');
var stepper = require('./stepper.js');
var PwmLED = require('./PwmLED.js');
var snipsUserName = 'YOUR_SNIPS_USERNAME';
var wakeword = 'hermes/hotword/default/detected';
var sessionEnd = 'hermes/dialogueManager/sessionEnded';
var lightState = 'hermes/intent/'+snipsUserName+':Lights';
client.on('connect', function() {
console.log('Connected to Snips MQTT server\n');
client.subscribe('hermes/hotword/default/detected');
client.subscribe('hermes/dialogueManager/sessionEnded');
client.subscribe(lightState);
});
client.on('message', function(topic,message) {
var message = JSON.parse(message);
switch(topic) {
// * On Wakeword
case wakeword:
everloop.startWaiting();
console.log('Wakeword Detected');
break;
// * On Light State Change
case lightState:
// Turn lights On/Off
try{
if (message.slots[0].rawValue === 'on'){
mosfet.lightsOn();
everloop.stopWaiting();
console.log('Lights On');
}
else if(message.slots[0].rawValue === 'off'){
mosfet.lightsOff();
everloop.stopWaiting();
console.log('Lights Off');
}
else if(message.slots[0].rawValue === 'increase brightness'){
mosfet.pwmup();
everloop.stopWaiting();
console.log('brightness increased');
} else if(message.slots[0].rawValue === 'decrease brightness'){
mosfet.pwmdown();
everloop.stopWaiting();
console.log('decrease brightness');
} else if(message.slots[0].rawValue === 'move left'){
stepper.left();
everloop.stopWaiting();
console.log('moved left');
} else if(message.slots[0].rawValue === 'move right'){
stepper.right();
everloop.stopWaiting();
console.log('moved right');
}
}
// Expect error if `on` or `off` is not heard
catch(e){
console.log('Did not receive any state')
}
break;
// * On Conversation End
case sessionEnd:
everloop.stopWaiting();
console.log('Session Ended\n');
break;
}
});

PwmLED.js incorporates the code required to control the LEDs on the MATRIX Creator.

var Gpio = require('pigpio').Gpio; 
var GatePins = [25]; 
var mosfet = [];
 
for (var i = 0; i<ledPins.length; i++) { 
	var led = new Gpio(ledPins[i], {mode: Gpio.OUTPUT});    
	leds.push(led);
}
 
setInterval(loop, 16); 
 
function loop() { 
 
	for(var i = 0; i<leds.length; i++) { 
 
		var led = leds[i]; 
		var brightness = Math.sin(((Date.now()/16)+(i*5))*0.2)*0.5 + 0.5; 
		brightness*=brightness*brightness; 
		brightness = Math.floor(brightness*255);
		mosfet.pwmWrite(brightness);
	}	
 
}

stepper.js enables movement of LED lamp with a stepper motor

var raspi = require('raspi-io');
var motor = require('stepper-drive');
var board = new stepper.Board({
   io: new raspi()
});

board.on("ready", function() {

  var stepper = new motor.Stepper({
    type: motor.Stepper.TYPE.DRIVER,
    stepsPerRev: 200,
     pins: {
     step: 11,
     dir: 13
    }
 });


stepper.rpm(180).ccw().accel(1600).decel(1600).step(2000, function() {

console.log("Done moving CCW");

// once first movement is done, make 10 revolutions clockwise at previously
//      defined speed, accel, and decel by passing an object into stepper.step
stepper.step({
  steps: 2000,
  direction: motor.Stepper.DIRECTION.CW
}, function() {
  console.log("Done moving CW");
});
});
});

mosfet.js includes the code required to turn the mosfet on and off. This mosfet will control power to the lights.

const matrix = require("@matrix-io/matrix-lite");
var methods = {};// Declaration of method controls at the end
var mosfetPin = 0;// The GPIO pin connected to mosfet gate pin
matrix.gpio.setFunction(0, "DIGITAL");
matrix.gpio.setMode(0,"output");
methods.lightsOff = function() {
   matrix.gpio.setDigital(0,"ON");
  console.log("Lights Have Been Turned Off");
};
methods.lightsOn = function() {
   matrix.gpio.setDigital(0,"OFF");
  console.log("Lights Have Been Turned On");
};
module.exports = methods;

Video demo

<add link here>