PIX3L PLOTT3R - Alexa Edition

PIX3L PLOTT3R – ALEXA EDITION

Background

Inspired by learning about the original dot-matrix printers, I decided to create one using LEGO MINDSTORMS. For this project, I decided to upgrade my plotter to be voice activated, using Alexa. Thus was born Pix3l Plott3r Alexa Edition. In this project, you can ask the Alexa device to print the image you want by saying, "Alexa, open Pixel Plotter. Print [picture name]". I was able to create a menu system to display on the Echo device. The menu allows the user to select what images to print. Taking advantage of the fact that Alexa is able to recite useful and related information, the user is also able to learn more about each image/destination.

Another major feature is to create an interactive hands-free photo booth. Since I was not able to access the built-in camera on the Echo Show using a custom skill. I decided to add my own webcam to the project. The webcam is connected to the MINDSTORMS EV3 USB port. This allows the user to take a photograph using the EV3 on Alexa's command and have it printed by the LEGO MINDSTORMS plotter. Imagemagick is used to process the picture, turning it into a line drawing that the printer can easily plot.

A user now has a complete Photo Booth and Printer available for use anywhere.

All photos and screenshots in this submission are taken by author and used with permission.

How it works:

Step 1: Ask Alexa to "Open Pixel Plotter"

Step 2: Select the image to learn more about from the menu

Step 3: Make sure paper is inserted into feeder

Step 4: Select image to print using Alexa

Or to use as a photo booth, pick Option 4, Custom Image.

Step 4: Ask Alexa to take a photograph

Step 5: Print your custom image

To watch the print process in action, view this YouTube video:

Robot Design Overview

The design uses one LEGO MINDSTORMS EV3 brick, one large motor, two medium motors and one color sensor. The large motor is used to lower and raise the pen that draws the image. A medium motor is used to feed the paper in. Another medium motor moves the entire print head left to right across the paper. The color sensor is used to detect the paper so that it is fed just the right amount. The design is kept simple and open so that it can be reproduced easily.

In addition, this project needs one webcam, an Amazon Echo and some thin markers.

Code Overview

The code consists of four major sections:

Part 1) The display manager that creates the list of pictures on the Alexa device;

Part 2) This part of the code handles common tasks like error logging, help requests, and communication with gadgets (not shown here, but provided in GitHub);

Part 3) The Alexa message receiver from Bluetooth on the EV3;

Part 4) The actual code that prints the image on the EV3-based printer.

Part 1:

For this part, the custom Alexa skill builder is used.

I begin by defining various pictures and descriptions for the print choices.

const data = [
{
ImageName: 'Taj Mahal', Abbreviation: 'tajmahal', Location: 'Agra, India', ImageYear: 1653, Description: 'The Taj Mahal is an ivory-white marble Islamic mausoleum on the south bank of the Yamuna river in the Indian city of Agra. It was commissioned in 1632 by the Mughal emperor Shah Jahan to house the tomb of his favourite wife, Mumtaz Mahal; it also houses the tomb of Shah Jahan himself. The tomb is the centrepiece of a 17-hectare complex, which includes a mosque and a guest house.',
},
{
ImageName: 'Statue of Liberty', Abbreviation: 'statueofliberty', Location: 'New York, NY', ImageYear: 1876, Description: 'The Statue of Liberty is a colossal neoclassical sculpture on Liberty Island in New York Harbor in New York, in the United States. The copper statue, a gift from the people of France to the people of the United States, was designed by French sculptor Frédéric Auguste Bartholdi and its metal framework was built by Gustave Eiffel. The statue was dedicated on October 28, 1886.',
},
{
ImageName: 'Gateway Arch', Abbreviation: 'saintlouis', Location: 'St. Louis, MO', ImageYear: 1958, Description: 'The Gateway Arch is a 630-foot monument in St. Louis, Missouri, United States. Clad in stainless steel and built in the form of a weighted catenary arch it is the world\'s tallest arch. Built as a monument to the westward expansion of the United States, the Arch, commonly referred to as "The Gateway to the West", is the centerpiece of Gateway Arch National Park and has become an internationally recognized symbol of St. Louis.',
},
{
ImageName: 'Custom', Abbreviation: 'custom', Location: 'Here', ImageYear: 2019, Description: 'Probably a picture of you. I hope you like it.'
},
];

These are displayed to the screen with a speech introduction:

handle: async function(handlerInput) {
  const request = handlerInput.requestEnvelope;
  let templateDirection = 'vertical';
  let templateNumber = 1;
  const imgHeight = [88];
  const imgWidth = [88];
...
  let speechOutput = `This is the main menu. Select the image to print by touch or say the image name.`;
  if (supportsDisplay(handlerInput)) {
    const imagesList = [];
    data.forEach((x) => {
      const image = new Alexa.ImageHelper().withDescription(`${x.Abbreviation}`);
      for (let y = 0; y < imgHeight.length; y += 1) {
        image.addImageInstance(getImageUrl(imgHeight[y], imgWidth[y], x.Abbreviation));
      }
      imagesList.push({
        token: x.Abbreviation,
        textContent: new Alexa.PlainTextContentHelper()
          .withPrimaryText(x.ImageName)
          .withSecondaryText(`Location: ${x.Location}`)
          .getTextContent(),
        image: image.getImage(),
      });
    });
    handlerInput.responseBuilder.addRenderTemplateDirective({
      type: `ListTemplate${templateNumber}`,
      token: 'listToken',
      backButton: 'hidden',
      title: `Welcome to Pix3l Plotter Alexa-edition`,
      listItems: imagesList,
    });
  }
  if (handlerInput.requestEnvelope.request.type === 'LaunchRequest') {
   speechOutput = welcomeMessage + speechOutput;
  }
  return handlerInput.responseBuilder
    .speak(speechOutput)
    .reprompt(repromptOutput)
    .getResponse();
  },
};

If the user said “print {picture}” the following print function is called:

const PrintIntentHandler = {
  ...
  if (endpointId === -1) {
    return handlerInput.responseBuilder
      .speak(`I couldn't find an EV3 Brick connected to this Echo device. Please check to make sure your EV3 Brick is connected, and try again.`)
      .getResponse();
  }

The function begins by giving a warning if the EV3 was not connected to the Alexa device.

Next, the payload is prepared to send to the EV3 with the type ‘print’ and the value as the picture name:

// Construct the directive with the payload containing 
  // the move parameters
  const directive = Util.build(endpointId, NAMESPACE, NAME_CONTROL,
    {
      type: 'print',
      picture: picture,
    });
  const speechOutput = (picture === "cancel")
    ? "Cancelling print"
    : `${picture} printing`;
  return handlerInput.responseBuilder
    .speak(speechOutput)
    .reprompt("awaiting command")
    .addDirective(directive)
    .getResponse();
  }
};

Part 2:

The util.js and common.js files (not shown here, but provided in GitHub) contain a variety of simple utility functions. For example, the common.js file contains the HelpIntentHandler that provides verbal directions to the user when requested and the ErrorHandler that tells the user when something has failed. The util.js contains code to simplify communication with Alexa gadgets. This util.js code is provided by Amazon in the Alexa+EV3 tutorials.

Part 3:

Here, I have the given Python code to interact with the Alexa API call the function print() with the input as the name of the picture said. The name of the inputted picture is stripped of spaces and made lowercase in order to remove the variations in the picture names the Alexa device reports (i.e. varied capitalization).

def _print(self, picture):
  #Take in the payload with the name
  print("Print command: ({})".format(picture), file=sys.stderr)
  print("printing "+picture.lower().replace(" ", "")+".png")

These lines determine if the picture reported is the name custom. If so, take a picture using the webcam. The linux program fswebcam is used to open the camera. Imagemagick is used to convert the picture to a line drawing, because the print quality looks better with lines. The picture is first converted to greyscale then the canny edge detection algorithm is used. The image must be downsampled to 120 pixels to print within the range of the plotter and it must be made monochrome (i.e. pure black or white pixels). The image is saved to custom.png.

# conjoin words and make lowercase to match filename
  if picture.lower().replace(" ", "") == "custom":
  # if the print id is "custom" - take a picture using the webcam.
  # Use imagemagick to do canny edge detection and various edits 
  # to optimize the picture for printing
  print("taking picture")
  os.system("fswebcam --no-banner -r 640x480 --jpeg 85 -D 1 web-cam-shot.jpg ; convert web-cam-shot.jpg -colorspace gray -canny 0x1+10%+10% -negate -rotate -90 -posterize 3 -resize 120 -threshold 90% -monochrome custom.png")

Below, we can see the effect in the following pictures:

Finally, the printer function, from the fourth part of the code, is called with the name of the picture+".png" to print.

# call printer() from printer.py
  print("starting print")
  printer.printer(picture.lower().replace(" ", "")+".png")

Part 4:

This section below is first part of the function printer(). It feeds in the paper until the color sensor at the far side recognizes it. It is then fed backwards a little bit to maximize print area.

def printer(filename):
  # move paper into feeder
  while col.value() < 50:
    paper.run_forever(speed_sp=1000)
  paper.run_to_rel_pos(position_sp=-7500, speed_sp=-1000, ramp_down_sp=500)
  waitformotor(paper)
  paper.stop()
  paper.reset()

Next, the Python imaging library is used to open the picture and analyze pixel by pixel. This portion simply opens it and flips the picture so that the printer does not print the reverse image:

img1 = Image.open(filename) #open image using python imaging library (PIL)
  img2=img1.convert("RGBA")
  img = img2.transpose(Image.FLIP_LEFT_RIGHT)
  width, height = img.size # get image size
  print(width," x ",height)

I then call the function processPic() is run to split the picture into channels and place the values into an array.

r_array, g_array, b_array, bl_array, e4col, lastRow = 
     processPic(img, width, height)

We can take a further look at the processPic function and how it works. The processing consists of a loop that iterates over each pixel in the image. At the last line in the beginning part of the loop below, the I extract the red, green, blue, and alpha/transparency (this should always be 0 but it is needed for PNG compatibility) levels of the pixel at point (w,h) into the variables r,g,b and a.

def processPic(img,width,height):
…
  while h != height:
    r_array.append([255]*width) # create an empty array
    g_array.append([255]*width)
    b_array.append([255]*width)
    bl_array.append([255]*width)
    while w >= 0:
      r,g,b,a = img.getpixel((w, h)) #get rgba of each pixel

The code below actually analyzes this pixel's color values and prints one of: "R" (for red), "G" (for green), "B" (for blue), "D" (for dark) and " " (for white). It also sets the appropriate array value corresponding to the pixel and the color.

# check if red, green, or blue is greatest in rgb 
      # values --- check if black or white also --> then append 
      # array differently for each switch case
      if r > g and r > b :
        e4col = true
        r_array[h][w] = 0
        lastRow = h
        print("R", end="")
      elif g > r and g > b :
        e4col = true
        g_array[h][w] = 0
        lastRow = h
        print("G", end="")
      elif b > r and b > g :
        e4col = true
        b_array[h][w] = 0
        lastRow = h
        print("B", end="")
      elif b < 50 and r < 50 and g < 50 :
        bl_array[h][w] = 0
        lastRow = h
        print("D", end="")
      else:
        print(" ", end="")
      w = w-1 #move to next pixel 
      # use -1 to flip image -> make images not 
      # backward when printed
  print(" "+str(h))
  w = width-1 #reset width counter
  h = h+1 #move to next row

Finally, the processed picture is returned as arrays to the caller.

return (r_array,g_array,b_array,bl_array,e4col,lastRow) # return arrays to caller

We can understand what the processPic function does by looking at a sample array of pixels from the picture split up into the RGBA channels values:

= [ ← ← ← ← ← ← ← ← ← ← ← ← ←  ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← w
h [(0,0,0,0),  (0,0,255,0), (0,255,0,0),(0,0,0,0),   (255,0,0,0)],
↓ [(0,0,0,0),  (0,0,255,0), (0,0,0,0),  (0,0,0,0),   (255,0,0,0)],
↓ [(0,255,0,0),(0,0,40,0),  (0,255,0,0),(10,10,20,0),(255,0,0,0)],
↓ [(0,0,0,0),  (0,0,255,0), (0,255,0,0),(0,0,0,0),   (255,0,0,0)],
↓ [(0,0,0,0),  (0,0,0,0),   (0,0,0,0),  (0,60,0,0),  (255,0,0,0)],
↓ [(0,0,0,0),  (0,0,255,0), (0,255,0,0),(0,0,0,0),   (0,0,0,0)],
↓ [(0,0,0,0),  (0,0,255,0), (0,255,0,0),(20,30,10,0),(255,0,0,0)]

The picture is essentially a 2 dimensional array. The above array is depicted nicely in the x and y dimensions similar the picture. The program first starts at h=0 (first row). Next, it looks at each entry in that row (each column). The columns are processed in a right to left direction (with a decreasing w variable, starting at w=width [of picture]) so that the printer plots on the paper in the same direction of the original picture (i.e. the axes are reversed compared to the picture). For each entry, it determines whether the red, green, or blue component is the greatest and add the presence of a respective red, green, or blue dot to their respective arrays. Black is defined as all components being less than 50 (dark grey included). In photography terms, this essentially posterizes the picture to the basic colors if there was previous downsampling. This is necessary since a felt pen is incapable of differentiating shading, greys, etc. Next, the row counter is increased to move to the next row and the process is repeated. The picture is also drawn out into the linux terminal so that the user gets a preview of the picture before printing.

A logged print out may look like the following:

If now continue looking at the printer function where we left off. After calling processPic, the printer function calls the runPrinter() function with arguments that pass the array of the picture’s color channel that we want printed. In this case, that color is black:

runPrinter(bl_array, width, lastRow+1) # print from array

Below, I show the details of the runPrinter function. This first of this function works similarly to the processPic function with the exception that it reads the pre-analyzed pixels of the passed color array. If the pixel array value is set it moves the pen to the determined position and makes a dot using the pen.

def runPrinter(array1,width,height):
  initial = time.time()
  xd = 0
  yd = 0
  xda = 0
  while yd < height:
    while xd < width:
      if array1[yd][xd] == 0: #is pixel black/colored?
        print("D", end="") # print block if black/colored pixel
        # move pen to dot's location
        head.run_to_abs_pos(position_sp=horiz_move*xd, 
          speed_sp=400, ramp_down_sp=500) 
        waitformotor(head)
        # lower and raise pen
        makedot(pen1,1) # print the dot
      else:
        print(" ", end="")
    xd = xd + 1
    xda = xda + 1
    # print percent of print complete and estimate remaining time
    print(" PCT: "+str(int(100*xda/(width*height)))+"% ; Time Remaining: "+str(int((100-100*xda/(width*height))*(time.time()-initial)/(100*xda/(width*height))))+"s") 
    yd = yd + 1
    xd = 0
    # move paper forward
    paper.run_to_abs_pos(position_sp=vert_move*(yd), 
      speed_sp=250,ramp_down_sp=500)
    # reset pen location
    waitformotor(paper)

We can understand this code more easily by looking at a sample array:

[ → → → → → → → → → → → → → → → → → →
↓ [0,255,0,0,255,255,255,0,255,0,255],
↓ [0,0,0,0,255,255,255,0,255,0,255],
↓ [0,255,255,0,255,255,255,0,255,0,255],
↓ [0,255,0,0,0,255,255,0,255,0,255],
↓ [0,0,0,0,255,255,0,0,255,0,255],
↓ [0,255,0,0,255,255,255,0,255,0,255],
↓ [0,255,0,255,255,0,255,0,255,0,0]
]

For each row, runPrinter checks each pixel and if the value is 0 (colored), then moves the print pen to (index of the pixel * horizontal move distance in degrees/pixel) and then prints a dot. It then moves to the next row by moving the paper feed motor to (pixel row id * vertical move distance in degrees/row). The move distances per pixel or row are calculated from the gear ratios. It repeats the process until the array is completely printed. An algorithm is used to determine the time remaining and percent complete based on how many rows of the total rows are printed. The print out to the terminal log looks something like this:

If the picture contains multiple colors, the runPrinter function is called to print the picture for each successive color channel:

resetMotors()
if e4col == true: # if the picture contains colored pixels, repeat with the colored arrays in the order red --> green --> blue
runPrinter(r_array, width, height)
resetMotors()
runPrinter(g_array, width, height)
resetMotors()
runPrinter(b_array, width, height)
resetMotors()

This allows the printer to support multicolor and black & white printing.

Debugging

To make the project easier to debug, I moved the code that checks for a connected EV3 from the LaunchHandler to the PrintIntentHandler. This allowed me to use the Developer Console Test system to try out my skill as shown below:

This proved extremely useful throughout the development process of the Alexa Skill portion of the overall application.

Recreating This Project:

The below instructions borrow from the hackster.io LEGO MINDSTORMS Voice Challenge instructions.

1. Follow the instructions at https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-setup-17300f to install ev3dev on your MINDSTORMS EV3 device and Visual Studio Code on your computer.

2. Follow the instructions to Register your Alexa Gadget at https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-setup-17300f

3. Download the code for this project from the linked GitHub repository. In the alexaplot.ini file, fill in the values for amazonId and alexaGadgetSecret with the values from step 2

4. Build the LEGO plotter model. Use either the build files in the "Building Instructions.zip" file from the GitHub repository or the lxf file (LEGO CAD File) linked to this project. To view the lxf file, you will need to download and install Lego Digital Designer from LEGO web site (https://www.lego.com/en-us/ldd)

5. This project uses an Amazon Skill. To create a skill, follow the instructions at https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-mission-3-4ed812#toc-create-your-alexa-skill-2. You should name your skill "Pixel Plotter" or any name you desire. This will be keyword to run the skill. Also, make sure to enable the Custom Interface Controller.

6. Next, you will need to define the skill interaction model. This is similar to the steps described in https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-mission-3-4ed812#toc-define-the-skill-interaction-model-4. In the Alexa Developer Console, under Interaction Model, click on JSON Editor. Cut and paste the model.json file provided in the linked GitHub repository (at pixelplotter-alexa/alexa/model.json) - replacing any existing content. After pasting the JSON into the Alexa skill JSON Editor, click Save Model, and then Build Model presented at the top of the console interface.

7. The skill also needs code to implement its logic. This is similar to the process described in https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-mission-3-4ed812?#toc-implementing-the-skill-logic-5. However, you will use the files in pixelplotter-alexa/alexa/lambda directory from GitHub instead of the files referenced in these linked instructions. First, click on Code in the top navigation bar of the Alexa Developer Console. Second, create a new file by clicking the New File icon in the upper-left of the Code Editor, and fill in the path and file name as /lambda/common.js. Third, copy the contents of index.js, package.json, util.js and common.js from the GitHub repository into the respective files of the Alexa skill - replacing the content of each file.

8. Click Deploy in the upper-right of the Alexa Skill Code Editor. Wait for the deployment process to complete.

9. Before you run the code on the EV3, you will need to install the Python Imaging Library (pillow). To do this, you will need to open an ssh connection to the EV3.

Right-click on the ev3dev device in the Explorer panel and select "Open SSH Terminal". This will get you a login that looks like:

In this window, you will need to run the following commands:

sudo apt-get update
sudo apt-get install -y python3-pip
sudo pip install pillow python-ev3dev

If you are prompted for a password, the default password for ev3dev is "maker"

10. Open VS Code and open the code folder from the GitHub repository. Using the instructions from step 1, you will need to "send your workspace" to the EV3 brick. Then using the EV3DEV DEVICE BROWSER part of the Explorer panel, right-click on the alexaplot.py file and choose Run from the displayed menu.

11. Once the program starts, you should hear your EV3 Brick play some tones, and see a prompt in the debug console telling you the friendly name of your gadget, which will look like Gadget000. This means your gadget is in pairing mode.

12. You're now ready to pair your EV3 Brick to your Echo device. This pairing process will depend on whether or not your Echo device has a screen. See these instructions for how to pair.

13, You should now be ready to go! Say "Alexa open Pixel Plotter". If you choose a different skill name in step 5, use that name instead of Pixel Plotter.