Unity AR with Arduino

Following from our previous post, where we created an Image Target and added a model to it (also allowing it to play music/video etc). In this tutorial, we can take it to another level by sending messages from Unity to Arduino using an ESP8266.

Prerequisites

Before you start this tutorial, please ensure you went through this tutorial, as there are a lot of key concepts required on Unity before proceeding forward.

Step 1: Create an Image Target

Open your previous Unity project you worked on, select New Scene, select Basic (Built-in) and save the scene as WithArduino in the Scenes folder.

Then in your new scene, delete the unity camera, and add the ARCamera, ImageTarget and Cube, notice how you don’t need to reimport anything as all previous assets can be reused in the new scene.

You would also noticed that you don’t need to add a license to your ARCamera as long as you already added a license in your previous scene. Hence, just click play and see if your cube spawns!

Step 2: Add scripts to your project

Select your Image Target and add a script, name it SendMessageBehavior.

Add the code below into the script, what this does is we are created a UDP script that allows us to send messages through the internet. (From desktop/phone to ESP8266 connected to your Arduino, more on that later)

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Net.Sockets;
using System.Net;
using System.Text;

public class SendMessageBehavior : MonoBehaviour
{

	private static SendMessageBehavior instance;

	public static SendMessageBehavior Instance
	{
		get { return instance; }
	}

	[Header("NODE MCU IP ADDRESS or COMPUTER IP ADDRESS")]
	public string serverIP;

	private const int PORT_NUM = 4445;
	private IPAddress serverAddr;
	private IPEndPoint endPoint;
	private Socket sock;
	private byte[] send_buffer;

	private void Awake()
	{
		instance = this;
	}

	void Start()
	{
		InitSocket();
	}

	void OnDisable()
	{
		//CloseSocket ();
	}

	void InitSocket()
	{
		//close socket in case its already open
		CloseSocket();
		//init socket
		sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
		sock.EnableBroadcast = true;
		serverAddr = IPAddress.Parse(serverIP);
		print(serverAddr);
		endPoint = new IPEndPoint(serverAddr, PORT_NUM);
	}

	void CloseSocket()
	{
		if (sock != null)
		{
			StopAllCoroutines();
			sock.Disconnect(true);
			try
			{
				sock.Shutdown(SocketShutdown.Both);
			}
			finally
			{
				sock.Close();
			}
		}
	}

	public void SendPacket(string message)
	{
		try
		{
			send_buffer = Encoding.ASCII.GetBytes(message);
			sock.SendTo(send_buffer, endPoint);
		}
		catch (SocketException s)
		{
			Debug.Log(s);
		}
		Debug.Log(message);
	}
}

Create another script on Image Target (name it LEDComm), this time we will override the image target OnTrackingFound() and OnTrackingLost() methods, and send a message to ESP8266 whenever the image target is detected or lost.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class LEDComm : DefaultObserverEventHandler
{
    protected override void OnTrackingFound()
    {
        if (mObserverBehaviour)
        {
            var rendererComponents = mObserverBehaviour.GetComponentsInChildren<Renderer>(true);
            var colliderComponents = mObserverBehaviour.GetComponentsInChildren<Collider>(true);
            var canvasComponents = mObserverBehaviour.GetComponentsInChildren<Canvas>(true);

            // Enable rendering:
            foreach (var component in rendererComponents)
                component.enabled = true;

            // Enable colliders:
            foreach (var component in colliderComponents)
                component.enabled = true;

            // Enable canvas':
            foreach (var component in canvasComponents)
                component.enabled = true;
        }
        SendMessageBehavior.Instance.SendPacket("on");
        OnTargetFound?.Invoke();
        Debug.Log("Turn on LED");
    }

    protected override void OnTrackingLost()
    {
        if (mObserverBehaviour)
        {
            var rendererComponents = mObserverBehaviour.GetComponentsInChildren<Renderer>(true);
            var colliderComponents = mObserverBehaviour.GetComponentsInChildren<Collider>(true);
            var canvasComponents = mObserverBehaviour.GetComponentsInChildren<Canvas>(true);

            // Disable rendering:
            foreach (var component in rendererComponents)
                component.enabled = false;

            // Disable colliders:
            foreach (var component in colliderComponents)
                component.enabled = false;

            // Disable canvas':
            foreach (var component in canvasComponents)
                component.enabled = false;
        }

        OnTargetLost?.Invoke();
        SendMessageBehavior.Instance.SendPacket("off");
        Debug.Log("Turn off LED");
    }
}

You should have two scripts on your Image Target, notice we have a Server IP (ESP8266 IP Address) to key in, but first we need to configure our ESP8266 first.

Step 3: Set up ESP8266 with Arduino

Materials needed:

• Arduino Uno x1 (microcontroller)
• ESP8266 ESP-01 x1
• DollaTek AMS1117-3.3 DC Voltage Regulator x1 (ESP8266 draws a lot of current for this project)
• Some wires
• Breadboard x1

Connect the components shown below, the white box is just a voltage regulator to step down Vin ( 5-12V) to 3.3v supplying sufficient current for the ESP8266.

Ensure you have a 2.4 GHz WiFi network you can connect to, otherwise create a mobile hotspot on your phone, and allow your laptop/ESP8266 to connect to it. Open Serial Monitor in Arduino, then type in the commands (the ones starting with AT) below, you should get an “OK” response.

AT

OK
AT+CWMODE_CUR=3

OK
AT+CWJAP="<WIFINAME>","<WIFIPASSWORD>"

OK
AT+CIFSR
+CIFSR:APIP,"192.168.4.1"
+CIFSR:APMAC,"1a:fe:34:f4:ef:b2"
+CIFSR:STAIP,"192.168.19.30"
+CIFSR:STAMAC,"18:fe:34:f4:ef:b2"

Record the STAIP, and paste it into the SERVER IP textbox in your Unity Scene. Note that the IP changes every time you connect to a new network. So please use the “AT+CIFSR” to double check. Play the Unity scene.

Type in the follow commands in the Serial Monitor, this should start a UDP server for the ESP8266 to listen for broadcast messages (meaning it will receive information from any devices connected to the hotspot)

AT+CIPMUX=1

OK
AT+CIPSERVER=1,80

OK
AT+CIPSTART=0,"UDP","0.0.0.0",4445,4445,2
0,CONNECT

Point your camera at an image you would like to track, you would notice the Arduino Serial Monitor printing messages when it detects/loses detection on the cube. “on” when detected, “off” when lost detection.