This is the smart home environment we created. It features some Assets from the Unity asset store, while others, such as the toaster and the kettle were designed by us in Blender. An EEG headset is connected to Unity, in order operate the P300 speller.
The P300 speller uses a reactive BCI paradigm. This means that the user is presented a stimuli, which they, and in turn, their brain activity react on. In this case it is by presenting a flashing image of a face over the images shown above. The user has to focus their attention on one of the symbols, and each time a face overlays their image, a positive ERP (response of the brain), can be seen in the brain activity around 300 milliseconds after the image flashed. The system works by flashing all the possible options (so randomly over the entire speller board). Once a response is detected in the brain, it check what image was being overlaid 300 seconds prior to the response.
Once the option is selected, it is send via User Data Protocol (UDP), to Unity, which in turn reacts to the given input.
The toast Icon makes toast pop up from the toaster in the scene. The kettle option triggers an animation of boiling water above the kettle. The light option turns the lights on and off within the scene. The trashcan icon makes the robot servant pick up the trash, and when near the trashcan throw it away.
The arrows on the speller board are used to move the Robot servant through the scene, which could be considered a form of telepresence.
Since this project is made within 1 sitting of 24 hours, it is still very much a proof of concept for the real world.
The nice thing about virtual reality is that it is fairly cheap (also in time) to create, while still showing what could be reality. Our vision for this project is for people who are unable to move, to have a robot servant that can be controlled without moving a muscle. This could allow people, that are paralyzed, or suffering from ALS to care for themselves more, and give them more freedom.
In the future we might want to connect this to augmented reality. Just imagine the possibilities of having a P300 overlay controlling a smart home for a real world environment!
Our reaction to winning the IEEE SMC 2021 Hackathon
This project was created during the Br41n.io Hackathon IEEE SMC 2021, and was our first project working with BCI. It was also the project that inspired us all to explore the possibilities of interacting with the biggest Enigma of all. The human Brain.