Building on last week’s progress, my group and I focused on incorporating seat pressure sensors, refining the RFID cards into stickers that could be discreetly placed beneath the offering bowls, and integrating our physical components with the Unity and Blender animations. Additionally, we experimented with Touch Designer to create dynamic lighting effects for our cabinets.
Partners: Queenie Hsiao and Michelle Xu
Since the cards didn’t integrate smoothly under the offering bowls, we decided to switch to RFID stickers, which could attach neatly under each bowl. This adjustment required only minor tweaks to the uid if-statements in the code.
Additionally, because we plan to place the bowls directly on the reader, the reader needed to lie flat. To achieve this, we extended the RFID reader as shown below. Currently, the extension coils are quite long but will be trimmed closer to the final fabrication stage. With the RFID reader now able to lie flat, we have greater flexibility in how the offering bowls with the RFID stickers interact with the reader.
Based on feedback from last week’s playtesting, we incorporated seat pressure sensors into our project. The intended user experience begins when the user sits on the chair, with the sensor placed beneath a cushion. By setting a low activation threshold, the sensor triggers easily when the user sits down, sending a message to Unity to initiate the starting sequence, welcoming the user to choose an offering.
Recording 2024-12-02 165758.mp4
The video above demonstrates the messages Unity receives when triggers are activated by the physical components (seat pressure sensor and RFID). It showcases an example where a user sits on the chair and then places the orange offering on the altar.
The code for Arduino and Unity have been revised to support continuous user interactions and fix an error caused by attempting to close an uninitialized or non-existent serial connection (sp). We also integrated the intro audio triggered by the user sitting on the seat pressure sensor, allowing us to test audio implementation within Unity.
/* Code for Seat Pressure Sensors and RFID Stickers */
#include <SPI.h>
#include <MFRC522.h>
#define RC522_SS_PIN 10 // The Arduino Nano pin connected to RC522's SS pin
#define RC522_RST_PIN 9 // The Arduino Nano pin connected to RC522's RST pin
String uid = "";
int lastSensorState = LOW;
int threshold = 512;
bool sensorHasBeenRead = false;
MFRC522 mfrc522(RC522_SS_PIN, RC522_RST_PIN);
void setup() {
Serial.begin(9600);
while (!Serial);
SPI.begin();
mfrc522.PCD_Init();
}
void loop() {
/* Pressure Sensor */
int sensorState = analogRead(A0);
if (sensorState >= threshold && !sensorHasBeenRead) {
if (lastSensorState < threshold) {
Serial.println("Sensor Triggered");
sensorHasBeenRead = true;
}
}
lastSensorState = sensorState;
/* RFID */
if (!mfrc522.PICC_IsNewCardPresent()) {
return;
}
if (!mfrc522.PICC_ReadCardSerial()) {
return;
}
if (mfrc522.uid.size && sensorHasBeenRead) {
for (int i = 0; i < mfrc522.uid.size; i++) {
uid += String(mfrc522.uid.uidByte[i], HEX);
}
if (uid == "a6a9e3f") {
Serial.println("orange");
} else if (uid == "1a6a9e3f") {
Serial.println("gold");
} else if (uid == "6a6a9e3f") {
Serial.println("lotus");
} else if (uid == "2a6a9e3f") {
Serial.println("apple");
} else if (uid == "3a6a9e3f") {
Serial.println("peach");
} else {
Serial.println("invalid offering");
}
sensorHasBeenRead = false;
}
delay(3000);
uid = "";
// commented out because we need the sensors and rfid readers to read continously
// mfrc522.PICC_HaltA(); // halt PICC
// mfrc522.PCD_StopCrypto1(); // stop encryption on PCD
}
/* Code for Unity animations and receiving values from Arduino */
using System;
using System.IO.Ports;
using UnityEngine;
public class FortuneTrigger : MonoBehaviour {
Animator animator;
public string portName = "COM3";
public int baudrate = 9600;
public static SerialPort sp;
public AudioSource playIntro;
public SendOSC osc;
public string debugMessage, offeringMessage, sensorMessage;
public int cabinetNumber;
private bool sensorHasBeenRead = false;
private string orangeVal = "orange";
private string goldVal = "gold";
private string lotusVal = "lotus";
private string appleVal = "apple";
private string peachVal = "peach";
private string orangeTrigger = "GenerateOrangeFortune";
private string goldTrigger = "GenerateGoldFortune";
private string lotusTrigger = "GenerateLotusFortune";
private string appleTrigger = "GenerateAppleFortune";
private string peachTrigger = "GeneratePeachFortune";
private string sensorTrigger = "SensorTrigger";
private int[] orangeCabinetNum = {10, 16, 6};
private int[] goldCabinetNum = {14, 15, 4};
private int[] lotusCabinetNum = {8, 11, 7};
private int[] appleCabinetNum = {13, 9, 2};
private int[] peachCabinetNum = {1, 3, 5};
void Start ()
{
animator = GetComponent<Animator>();
sp = new SerialPort(portName, baudrate, Parity.None, 8, StopBits.One);
OpenConnection();
osc = GameObject.FindGameObjectWithTag("OscController").GetComponent<SendOSC>();
}
void Update() {
string sensorValue, offeringValue;
System.Random RNG = new System.Random();
int cabinetArrIndex = RNG.Next(0, 3);
if (sp.IsOpen)
{
try {
if (sp.BytesToRead > 0) {
if (!sensorHasBeenRead) {
sensorValue = sp.ReadLine().Trim();
if (sensorValue == "Sensor Triggered") {
animator.SetTrigger(sensorTrigger);
sensorMessage = sensorValue;
Debug.Log("Sensor Value: " + sensorValue);
// when the sensor is triggered, the starting intro audio should play
**playIntro.Play();**
sensorHasBeenRead = true;
}
} else {
offeringValue = sp.ReadLine().Trim();
Debug.Log("Offering Value: " + offeringValue);
HandleOfferingValue(offeringValue, cabinetArrIndex);
}
}
}
catch (TimeoutException ex)
{
Debug.LogWarning("Serial read timeout: " + ex.Message);
}
catch (Exception ex)
{
Debug.LogError("Error reading from serial port: " + ex.Message);
}
}
}
private void HandleOfferingValue(string offeringValue, int cabinetArrIndex)
{
if (offeringValue == orangeVal)
{
animator.SetTrigger(orangeTrigger);
cabinetNumber = orangeCabinetNum[cabinetArrIndex].ToString();
Debug.Log("Generating Orange Fortune");
}
if (offeringValue == goldVal)
{
animator.SetTrigger(goldTrigger);
cabinetNumber = goldCabinetNum[cabinetArrIndex].ToString();
Debug.Log("Generating Gold Fortune");
}
if (offeringValue == lotusVal)
{
animator.SetTrigger(lotusTrigger);
cabinetNumber = lotusCabinetNum[cabinetArrIndex].ToString();
Debug.Log("Generating Lotus Fortune");
}
if (offeringValue == appleVal)
{
animator.SetTrigger(appleTrigger);
cabinetNumber = appleCabinetNum[cabinetArrIndex].ToString();
Debug.Log("Generating Apple Fortune");
}
if (offeringValue == peachVal)
{
animator.SetTrigger(peachTrigger);
cabinetNumber = peachCabinetNum[cabinetArrIndex].ToString();
Debug.Log("Generating Peach Fortune");
}
// Send the cabinet number to the osc script to communicate with TouchDesigner
osc.SendInteger(cabinetNumber);
offeringMessage = offeringValue;
sensorHasBeenRead = false;
}
public void OpenConnection()
{
if (sp != null)
{
if (sp.IsOpen)
{
debugMessage = "Closing port!";
}
else
{
sp.Open();
if (sp.IsOpen)
{
Debug.Log("Just opened");
}
sp.ReadTimeout = 20;
debugMessage = "Port Open!";
}
}
else
{
if (sp.IsOpen)
{
print("Port is already open");
}
else
{
print("Port == null");
}
}
}
void OnApplicationQuit()
{
sp.Close();
}
}
During last week’s playtest, we received feedback on enhancing how we guide the user’s attention from the altar to the cabinet, beyond simply displaying the cabinet number on the screen. One suggestion was to use projection mapping with TouchDesigner to illuminate the specific cabinet the user needs to open based on the generated cabinet number.
With some help from Michael, Queenie got the basics of TouchDesigner working!
We consulted with many people to understand the complexities and process of establishing communication between Unity, Arduino, and TouchDesigner. Initially, we considered abandoning the idea of linking Unity with TouchDesigner altogether and even doubted whether it would be possible to send cabinet number information to TouchDesigner. However, we eventually got in touch with individuals who had experience using the OSC library to facilitate communication between Unity and TouchDesigner, which provided the foundation we needed to move forward.
Using the UnityOSC library, we developed a separate script (SendOSC) and created a new GameObject in Unity linked to the OSC script and library components. We then integrated the FortuneTrigger script, which calculates the cabinet number, with the OSC script by implementing a function to pass this cabinet value to TouchDesigner.