This tutorial is thought to be the last stage where I aim to build an Augmented Reality & IoT program.
At this stage we will take data from a MQTT broker (Adafruit IO io.adafruit.com) to show it in an Android App built using Unity and Vuforia.
For this stage I'm taking as reference the following YouTube video that I find very helpful
I've chosen Unity because it is one of the most used Augmented Reality softwares, because it already has mqtt libraries developed and because it can interact with Vuforia.Project structure
Adafruit IO is the MQTT broker that receives information from an IoT device.
Unity is the platform that will take the information from Adafruit IO and show it in a digital environment. It will also help us to build the App that we will run in our mobiles or tablets.
Vuforia is the the software that recognizes and tracks target images in a digital environment.
Therefore, the project at is core is run using Unity that connects to Adafruit IO and uses Vuforia functionalities.What You'll Need
You'll need the following:
- An account at Adafruit IO (io.adafruit.com)
- Unity software (https://unity.com/download) I'm using 2021.2.16f1 version
- M2Mqtt libraries for Unity (https://github.com/CE-SDV-Unity/M2MqttUnity)
- Vuforia software (https://developer.vuforia.com) I'm using 10.5.5 version
Log in into your Vuforia account and click on Develop, Target Manager and Add database. Give it a representative name for your project and select the Device option to create the Database.
The Database will store every target image that you will have in your project.
Now, select the recently created Database and then on Add Target.
In our case the target will be a Single Image, so select that option and then select an image file from the browser. In my case I selected an image with a QR code that I created for this project. Assign a representative name for that image that will later be used in Unity. Once that you have added the target images, click on Download Database and then in Database for Unity Editor.
Last but not least, still in Vuforia webpage click on License Manager, assign a name to your project, check the box and confirm to get your License Key. That Key will allow us to run Vuforia functionalities on Unity.
That's it, you already have all the documents and the License Key that you need from Vuforia.Setting up Vuforia in Unity
Once that you have downloaded Unity, install and run the software. If you are going to run a mobile or tablet app you should also install its extensions in this step. In my case I installed the extensions for an Android App.
In the main page of UnityHub select New Project, then 3D Core option and assign a representative name to it. In this case, I named it as I named the Vuforia one "AR_IoT_QR"
Drag and drop M2Mqtt and M2MqttUnity folders downloaded from Github to the Assets of the project as shown in the image below.
Then, create two new folders inside the Assets file where you will store the main scripts that will run your program. In my case I called those folders CE and Scripts
Inside of Scripts drag and drop mqttReceiver.cs and mqttController.cs files. You can find both scripts attached with this project.
In the left side of the screen click the right button and create an empty Game Object, name it MQTT_Receiver and add the script mqttReceiver.cs dragging the script into it. Then, complete the fields on the right side of the screen.
- Broker Address (in this case is io.adafruit.com)
- Check on Auto Connect
- Mqtt User Name (your MQTT broker username, in my case is lesurfe)
- Mqtt Password (your MQTT broker password, in my case is Adafruit IO Key)
- Topic suscribe (its the route to the MQTT broker feed which was added to M2MQTTUnityTest in the previous stage, in my case was lesurfe/feeds/text)
Create a new GameObject, name it ControlledObject and drag and drop mqttController into it. Then click on Event Sender tab and select MQTT_Receiver.
Lastly, create a 3D Object - Text Mesh Pro and drag and drop mqttController.cs into it.
In this case you'll need to edit the script by clicking in the three vertical dots at the right of Mqtt Controller as we did in the previous stage. Erase the whole content of the scrip and paste the following:
public class mqttController : MonoBehaviour
public string nameController = "Controller 1";
public mqttReceiver _eventSender;
_eventSender.OnMessageArrived += OnMessageArrivedHandler;
private void OnMessageArrivedHandler(string newMsg)
Debug.Log("Event Fired. The message, from Object " +nameController+" is = " + newMsg);
Now go to your Downloads folder and open Vuforia to Unity package. Once that is installed, create an AR camera by clicking on the plus symbol, Vuforia Engine and AR camera as shown in the image below.
Once that you've created an AR camera object, now right click over the AR camera object and select Vuforia Engine and Image Target. Continue by opening your projects database that you've also downloaded from Vuforia and importing it to Unity. Finally, select AR camera object and in the right side of your screen click on Open Vuforia Engine configuration to paste the licence Key in the Add license Key box as shown in the image below.
Lastly, go to Image target, select Image Target Behaviour, type From Database, select the respective project and the target image that you want to use as target.
To display the image into Unity simply double click on Image Target Object.
You can set different image views by playing with the box and the coordinates at the right top corner of the image.Setting up Unity
We've already set the structure where the software will track the target image.
To design the digital environment we will start by scaling the target image to X=50, Y=50 and Z=50. We are doing this for practical reasons during the design, once that we've finished with the settings we then need to set the scale back to 1.
Now we are going to create a dashboard where Unity will display the desired variables to control. I designed one to display temperature and time values. You can find the image attached with this project.
Right click on Image Target Object and selecting 3D Object and then Plane
Then drag and drop the dashboard image into Resources and then drag and drop it again but this time to the Plane Object
You can then manipulate the image as you like, in my project I liked how it looked like emerging from the image so I needed to rotate it 180 degrees in the y-axis and 90 degrees in the x-axis (but in a later step).
To modify the size of the dashboard we simply click on the Rect tool (square button) and change the size to an according one taking as reference the QR code. You can also shift the location of the dashboard image by moving the image using the Move tool (arrows button)
After a few modifications I finished with the image as I wanted.
Now we will continue by locating the Text (TMP) objects that we want to display in the Dashboard image.
Once that you've located them drag them into the Image Target Object, this will help you to re-size everything in one shot.
One important thing that I noticed is that if you are aiming to display more than one variable at the same time you'll have to create every variable from scratch.
Do not copy and paste an existing variable to create a new one because the program will only display the original variable and it won't display the copy and paste ones.
Once that you've set the digital environment re-scale the Image Target to X=1, Y=1 and Z=1 and save the project.
If you press PLAY button and then Scene you should see how the Text (TMP) Objects start showing your MQTT inputs.Creating an Android App
You are almost there! The last step is creating your Android App. To do so, go to File and then to Build Settings.
Then select the Operating System where you are going to run your app (you should see Unity logo at its side!) and finally press Build.
In my case I've chosen Android. Once that the .apk file has been created select a place to save it. The .apk file will later be transferred to your phone to run the AR & IoT app.
While building your project you might come up with this building error.
To solve it you need to change the Target API from Target API Level field in the Player Settings window. You can find the Target API Level scrolling down until you find Identification in Other Settings. You'll need to select the same Android version according that your mobile/tablet has.
After doing that change you should be able to build your app without issues.
Finally, you'll need to plug-in your device and drag and drop the .apk file into the Downloads folder.
Now take your device, open Downloads folder and select the .apk file to install your app. Once that it has been installed you should see the app in your main Menu.
If you open that app probably your Operating System will ask you if you approve opening an app created by an unknown company, you should approve that and if later asks you for a Play Protect analysis you should skip it.
Congratulations! You are now ready to start using your Augmented Reality and IoT app!