Unity 5 and the recording of pulse rate with the Arduino.

One of my tasks this Trimester is to get feedback and use it, from something other than standard game controllers or the mouse.

I have teamed up with designer, Savik Fraguela (yet again), and I am providing bio-feedback for his game.  In order to keep it inexpensive at this stage, I am using feedback obtained from an Arduino board and software that comes with the “Pulse Sensor” from http://www.pulsesensor.com.

This meant that I needed to download and install several Software suites and the FDTI drivers.  The drivers were a pain in my case because I am using Windows 8 and this operating system hates using unverified, or unsigned drivers.  I had to get past that by following the directions on this site : https://learn.sparkfun.com/tutorials/disabling-driver-signature-on-windows-8

Installing the rest of the software was uneventful as was attaching the monitor to the Arduino board.

I was able to get the software (called a sketch in Arduino Studio) running and pushed onto the board.  I was also able to get the serial driver showing what feedback was comint through the port.

My first problem came when trying to write my Unity script.  I eventually had errors saying that the serial was not open.  By this stage, I had to return the Arduino hardware to my Lecturer, but I suspected that it was because ArduinoStudio was using the port to read the information.

This was confirmed over the weekend.  I again ran Arduino Studio but didn’t display the Serial Port in that suite.  Unity had no problems with the set up.

Now the fun part.  I am used to manipulating gameobjects, and not so much strings and byte[] and char[].

I was trying to put the data from the serial port into a char array and specifying the length as 5, because the data being sent was a Letter that indicated what sort of information was being sent, then an int of 2 or 3 characters and then a carriage return.  I was able to display only the numbers and nothing else.  The code I was using was serialPort.ReadByte().

I eventually found (thanks Google) that I should be reading using serialPort.ReadLine() and copying that to a string.  This allowed me to read the whole line of data coming over the serial port in the format of S203, etc.

The incoming data was split into three categories.  Data that starts with an S prefix which gives the user raw data.  Data with a Q prefix that is apparently some sort of time measurement from the last beat and data that has a B prefix, which is the one I wanted as it is the beats per minute.  This is taken over 10 beats and averaged out for a minute.

c# strings are so easy to work with.  I thought that I would have to split the string down into a char array and try to work with that, but as it turns out, a string is already a char array.  Each element can be dealt with as string[0], string[1], etc.  That was fine until I tried to use this to find out the start of the string .. if(string[0] == “B”).  I was told that Unity couldn’t compare a string with a char.

More searching and I found that you could do this if(string.StartsWith(“B”)).  This has me beat.  It sounds the same as if(string[0] == “B”), but Unity accepted that and it worked.  I was now getting all of the data that had the B prefix.  I did notice that the information seemed to be fluctuating pretty steadily, and occasionally wildly.

Because there was no guarantee as to how long the ints were, they could be 2, 3, or 4 numbers long(although in the case of the B prefix, it is fair to assume that they will be only 2 or 3 numbers long) I made use of string.Length, they are arrays after all.

As only the B data is coming through to this point, I then created a new temp string and appended it with the contents of my original string, excluding the prefix.  I then converted that temp string into an int.

In order to try and settle down the fluctuations, I created a List of ints and when the count reached 5, it would take the average of those bpm and return this new bpm. (after removing the first member of the List)

I know that Savik will be using an enum to gauge the factor of how scared the player is and using that enum to ramp up the fear inducing elements of his game.  If this list isn’t stable enough, I will look at making the list longer and even consider taking the highest and lowest values out, then averaging the rest and using that information for the new bpm.

In conjunction with this, we were considering using OpenCV and OpenCVSharp (and even some other formats) to use facial tracking from a web cam and supplement information from them to aid in the bpm.  I will write another blog about this regardless of whether I can get it working in Unity 5, so that someone can see the effort I have put into trying to get it working and perhaps show me if and where I went wrong.  I did find a project that will work in Unity 4, but there seems to be a problem with Unity 5 reading the haars_Cascade_facial .xml file.

In the mean time, here is the file that I used to get the pulse monitor working with in Unity:

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System.IO.Ports;
using System.IO;

public class SerialListener : MonoBehaviour
{
public SerialPort serial = new SerialPort(“COM3”, 115200);//create new serial port
private string beat = “”;// string to hols the data in
private List<int> beatPM = new List<int>(); //List to hold the beats

void Start ()
{
//open a new connection
OpenConnection();
}

void Update ()
{
beat = “”;
//Read data input … should be a letter to start (char)
//then 3 numbers and then carriage return
beat = serial.ReadLine();
//        print (beat);
//Create an int that can be used by the Client
int somethingToSend = AnalyseBeats(beat);
print (somethingToSend);

}

void OnApplicationQuit()
{
serial.Close();
}

//Function connecting to Arduino
public void OpenConnection()
{

if (serial != null)
{
if (serial.IsOpen)
{
serial.Close();
print(“Closing port, because it was already open!”);
//message = “Closing port, because it was already open!”;
}
else
{
serial.Open();  // opens the connection
serial.ReadTimeout = 1000;  // sets the timeout value before reporting error
print(“Port Opened!”);
//        message = “Port Opened!”;
}
}
else
{
if (serial.IsOpen)
{
print(“Port is already open”);

}
else
{
print(“Port == null”);
}
}
}
int AnalyseBeats(string str)
{
//Data comes in with a “S” header, a “Q” header or a “B” header
//S = raw data .. “Q” is (allegedly) the time since the last beat was detected
//and B is the one that we want.  It is the beats per minute averaged over 10 beats

if (str.StartsWith(“B”))
{
//            print (“Original data.” + str);
//create a new string
string t = null;
//append to t everything except the starting letter
for(int j = 1; j<str.Length; ++j)
{
t += str[j];
}
//Convert that string into an int
int temp = int.Parse(t);
//            print (“Modified data.” + temp);
//add that int to the list
beatPM.Add(temp);

}
//        print (“BMP.count.” + beatPM.Count);

//Create an average of the BPM to give a more stable reading of the current pulse rate
if(beatPM.Count >=5)
{
int bAv = 0;
for (int i = 0; i < beatPM.Count; ++i)
{
bAv += beatPM[i];
}
bAv= (int) bAv/beatPM.Count;
//            Debug.Log(“Average over last 5 beats = ” + bAv);
beatPM.RemoveAt(0);
return bAv;
}

return 0;

}
}

Advertisements

2 thoughts on “Unity 5 and the recording of pulse rate with the Arduino.

  1. Michael Knight

    Right on this is pretty phenomenal work. I would love to somehow implement some of what you have done here into my Knight Rider Dash software project. This would possibly go some ways towards making K.I.T.T. able to actually read some vital signs. Not sure if you know what I mean if you’re not a fan of the 80’s TV show Knight Rider 😉
    I would love to hear your thoughts and feedback on that idea as I could probably use your help with something like that.

    All the best with the project and I’m looking forward to where you take it.

    Michael

    Like

    Reply
    1. ironstein62 Post author

      Thanks for the interest Michael. The biggest problem I see with using the arduino hardware was the fact that you lost a thumb for the pulse monitor. It seems to have the stronger pulse measurement. I didn’t try with the ear lobe (an alternative site) because the lead wasn’t long enough, but I would tend to think that the feedback from the earlobe wouldn’t be as reliable. The ideal device would be a heart rate monitor that went around the chest and could also monitor breathing, but again, this would be unweildly for use in a game and could be cost prohibitive. You are welcome to fiddle with my code as it is very basic and would occur to anyone who had to do a little research. Actually, I suspect that I might have had to make some last minute changes … I will have a look and if there were, I can upload the latest version. I was only responsible for getting this data into the project for the designer. I can also try and find the game site, but I think he is charging a token fee for the download. Let me know what specifically you would like to know about and I will answer, if I can 🙂
      Cheers
      Richard

      Like

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s